2024-01-23T08:25:19.345 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-23T08:25:19.345 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --owner scheduled_cbodley@teuthology --archive /home/teuthworker/archive/cbodley-2024-01-22_19:46:51-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7527396 --verbose --name cbodley-2024-01-22_19:46:51-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi -- /home/teuthworker/archive/cbodley-2024-01-22_19:46:51-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7527396/orig.config.yaml 2024-01-23T08:25:19.399 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-23T08:25:19.488 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/cbodley-2024-01-22_19:46:51-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7527396 branch: wip-yuri8-testing-2024-01-18-0823 description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: null first_in_suite: false job_id: '7527396' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: cbodley-2024-01-22_19:46:51-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri8-testing-2024-01-18-0823 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 0edf41a622739843d6b978b179ff3227b476dd9d ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 0edf41a622739843d6b978b179ff3227b476dd9d kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri8-testing-2024-01-18-0823-plus-55233 sha1: 9db6bac5568c8be6cfd98da20bd2b62582f0776f owner: scheduled_cbodley@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 6631 sha1: 0edf41a622739843d6b978b179ff3227b476dd9d sleep_before_teardown: 0 subset: 111/12000 suite: fs suite_branch: wip-yuri8-testing-2024-01-18-0823-plus-55233 suite_path: /home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa suite_relpath: qa suite_repo: https://github.com/cbodley/ceph.git suite_sha1: 9db6bac5568c8be6cfd98da20bd2b62582f0776f targets: smithi059.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFnPecS4qMynd+IIA/wWoEbFcLUsPHz/OCadDhe+id0B+IvoB2IfSE2xoqvZNe3ZL856Tnc42YnsRd/eOSxzGIc= smithi137.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMzyUqHuzAHFsSqL7zeDd6gRwxbOLdPxiEfr+GvgX1i0VlB597qQ/Sg8Xn1J9zN3+m7vLflnL1OsnpeMFpcMQ0I= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-22_19:46:51 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: cbodley verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-01-23T08:25:19.488 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa; will attempt to use it 2024-01-23T08:25:19.489 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks 2024-01-23T08:25:19.489 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-23T08:25:19.491 INFO:teuthology.task.internal:Checking packages... 2024-01-23T08:25:19.513 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '0edf41a622739843d6b978b179ff3227b476dd9d' 2024-01-23T08:25:19.513 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-23T08:25:19.513 INFO:teuthology.packaging:ref: None 2024-01-23T08:25:19.513 INFO:teuthology.packaging:tag: None 2024-01-23T08:25:19.513 INFO:teuthology.packaging:branch: wip-yuri8-testing-2024-01-18-0823 2024-01-23T08:25:19.513 INFO:teuthology.packaging:sha1: 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-23T08:25:19.513 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri8-testing-2024-01-18-0823 2024-01-23T08:25:19.819 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-766.g0edf41a6 2024-01-23T08:25:19.820 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-23T08:25:19.827 INFO:teuthology.task.internal:no buildpackages task found 2024-01-23T08:25:19.827 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-23T08:25:19.889 INFO:teuthology.task.internal:Saving configuration 2024-01-23T08:25:19.907 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-23T08:25:19.914 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-23T08:25:19.937 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/cbodley-2024-01-22_19:46:51-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7527396', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-23 08:20:06.787289', 'locked_by': 'scheduled_cbodley@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFnPecS4qMynd+IIA/wWoEbFcLUsPHz/OCadDhe+id0B+IvoB2IfSE2xoqvZNe3ZL856Tnc42YnsRd/eOSxzGIc='} 2024-01-23T08:25:19.958 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/cbodley-2024-01-22_19:46:51-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7527396', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-23 08:20:06.785801', 'locked_by': 'scheduled_cbodley@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMzyUqHuzAHFsSqL7zeDd6gRwxbOLdPxiEfr+GvgX1i0VlB597qQ/Sg8Xn1J9zN3+m7vLflnL1OsnpeMFpcMQ0I='} 2024-01-23T08:25:19.958 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-23T08:25:19.965 INFO:teuthology.task.internal:roles: ubuntu@smithi059.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-01-23T08:25:19.965 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-01-23T08:25:19.965 INFO:teuthology.run_tasks:Running task console_log... 2024-01-23T08:25:20.047 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2faa53d160>, signals=[15]) 2024-01-23T08:25:20.048 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-23T08:25:20.056 INFO:teuthology.task.internal:Opening connections... 2024-01-23T08:25:20.056 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi059.front.sepia.ceph.com 2024-01-23T08:25:20.058 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T08:25:20.166 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi137.front.sepia.ceph.com 2024-01-23T08:25:20.167 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T08:25:20.256 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-23T08:25:20.265 DEBUG:teuthology.orchestra.run.smithi059:> uname -m 2024-01-23T08:25:20.283 INFO:teuthology.orchestra.run.smithi059.stdout:x86_64 2024-01-23T08:25:20.283 DEBUG:teuthology.orchestra.run.smithi059:> cat /etc/os-release 2024-01-23T08:25:20.340 INFO:teuthology.orchestra.run.smithi059.stdout:NAME="CentOS Stream" 2024-01-23T08:25:20.341 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION="9" 2024-01-23T08:25:20.341 INFO:teuthology.orchestra.run.smithi059.stdout:ID="centos" 2024-01-23T08:25:20.341 INFO:teuthology.orchestra.run.smithi059.stdout:ID_LIKE="rhel fedora" 2024-01-23T08:25:20.341 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_ID="9" 2024-01-23T08:25:20.341 INFO:teuthology.orchestra.run.smithi059.stdout:PLATFORM_ID="platform:el9" 2024-01-23T08:25:20.341 INFO:teuthology.orchestra.run.smithi059.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-23T08:25:20.341 INFO:teuthology.orchestra.run.smithi059.stdout:ANSI_COLOR="0;31" 2024-01-23T08:25:20.341 INFO:teuthology.orchestra.run.smithi059.stdout:LOGO="fedora-logo-icon" 2024-01-23T08:25:20.341 INFO:teuthology.orchestra.run.smithi059.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-23T08:25:20.341 INFO:teuthology.orchestra.run.smithi059.stdout:HOME_URL="https://centos.org/" 2024-01-23T08:25:20.342 INFO:teuthology.orchestra.run.smithi059.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-23T08:25:20.342 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-23T08:25:20.342 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-23T08:25:20.343 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2024-01-23T08:25:20.373 DEBUG:teuthology.orchestra.run.smithi137:> uname -m 2024-01-23T08:25:20.390 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2024-01-23T08:25:20.390 DEBUG:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2024-01-23T08:25:20.448 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="CentOS Stream" 2024-01-23T08:25:20.448 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="9" 2024-01-23T08:25:20.448 INFO:teuthology.orchestra.run.smithi137.stdout:ID="centos" 2024-01-23T08:25:20.448 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE="rhel fedora" 2024-01-23T08:25:20.448 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="9" 2024-01-23T08:25:20.449 INFO:teuthology.orchestra.run.smithi137.stdout:PLATFORM_ID="platform:el9" 2024-01-23T08:25:20.449 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-23T08:25:20.449 INFO:teuthology.orchestra.run.smithi137.stdout:ANSI_COLOR="0;31" 2024-01-23T08:25:20.449 INFO:teuthology.orchestra.run.smithi137.stdout:LOGO="fedora-logo-icon" 2024-01-23T08:25:20.449 INFO:teuthology.orchestra.run.smithi137.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-23T08:25:20.449 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://centos.org/" 2024-01-23T08:25:20.449 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-23T08:25:20.449 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-23T08:25:20.449 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-23T08:25:20.449 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2024-01-23T08:25:20.472 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-23T08:25:20.483 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-23T08:25:20.504 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-23T08:25:20.504 DEBUG:teuthology.orchestra.run.smithi059:> test '!' -e /home/ubuntu/cephtest 2024-01-23T08:25:20.507 DEBUG:teuthology.orchestra.run.smithi137:> test '!' -e /home/ubuntu/cephtest 2024-01-23T08:25:20.523 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-23T08:25:20.530 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-23T08:25:20.531 DEBUG:teuthology.orchestra.run.smithi059:> test -z $(ls -A /var/lib/ceph) 2024-01-23T08:25:20.564 DEBUG:teuthology.orchestra.run.smithi137:> test -z $(ls -A /var/lib/ceph) 2024-01-23T08:25:20.660 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-23T08:25:20.715 INFO:teuthology.run_tasks:Running task kernel... 2024-01-23T08:25:20.730 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-23T08:25:20.731 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-23T08:25:20.731 DEBUG:teuthology.orchestra.run.smithi059:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-23T08:25:20.731 DEBUG:teuthology.orchestra.run.smithi137:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-23T08:25:20.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-23T08:25:20.748 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-01-23T08:25:20.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-23T08:25:20.749 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2024-01-23T08:25:20.806 INFO:teuthology.orchestra.run.smithi137.stdout:5.14.0-361.el9.x86_64 2024-01-23T08:25:20.806 INFO:teuthology.task.kernel:Running kernel on smithi137: 5.14.0-361.el9.x86_64 2024-01-23T08:25:20.806 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum install -y kernel 2024-01-23T08:25:20.807 INFO:teuthology.orchestra.run.smithi059.stdout:5.14.0-361.el9.x86_64 2024-01-23T08:25:20.807 INFO:teuthology.task.kernel:Running kernel on smithi059: 5.14.0-361.el9.x86_64 2024-01-23T08:25:20.807 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum install -y kernel 2024-01-23T08:25:23.285 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - AppStream 9.5 MB/s | 7.8 MB 00:00 2024-01-23T08:25:23.322 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS-9 - AppStream 9.0 MB/s | 7.8 MB 00:00 2024-01-23T08:25:25.061 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.5 MB 00:00 2024-01-23T08:25:25.281 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.5 MB 00:00 2024-01-23T08:25:26.226 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-01-23T08:25:26.246 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-01-23T08:25:30.006 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - AppStream 7.1 MB/s | 19 MB 00:02 2024-01-23T08:25:33.593 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - AppStream 3.0 MB/s | 19 MB 00:06 2024-01-23T08:25:33.964 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - CRB 16 MB/s | 5.9 MB 00:00 2024-01-23T08:25:35.481 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 15 kB 00:00 2024-01-23T08:25:36.442 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 25 MB/s | 20 MB 00:00 2024-01-23T08:25:39.103 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 5.9 MB 00:01 2024-01-23T08:25:40.570 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 15 kB 00:00 2024-01-23T08:25:41.366 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-01-23T08:25:42.049 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 15 MB/s | 20 MB 00:01 2024-01-23T08:25:43.534 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:25:43.534 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:25:43.534 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:25:43.534 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.534 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:25:43.534 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.534 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:25:43.534 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.534 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:25:43.535 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.535 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:25:43.535 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.535 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:25:43.535 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.535 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:25:43.535 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.535 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:25:43.535 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.535 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:25:43.535 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.535 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:25:43.536 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.536 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:25:43.536 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.536 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:25:43.536 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.536 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:25:43.536 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.536 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:25:43.536 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.536 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:25:43.536 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.537 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:25:43.537 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.537 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:25:43.537 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.537 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:25:43.537 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.537 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:25:43.537 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.537 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:25:43.537 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.537 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:25:43.538 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.538 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:25:43.538 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.538 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:25:43.538 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.538 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:25:43.538 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.538 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:25:43.538 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.538 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:25:43.538 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.538 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:25:43.538 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.538 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:25:43.538 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.539 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:25:43.539 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.539 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:25:43.539 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.539 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:25:43.539 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.539 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:25:43.539 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.539 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:25:43.539 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.539 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:25:43.539 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.539 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:25:43.539 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.539 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:25:43.540 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.540 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:25:43.540 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.540 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.540 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.540 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:25:43.540 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.541 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.541 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.541 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.541 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:25:43.541 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.541 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.541 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.541 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.541 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.542 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.543 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:25:43.543 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.543 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.543 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.544 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.544 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:25:43.544 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.544 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.544 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.544 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.544 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:25:43.544 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.544 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.544 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.544 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.545 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:25:43.545 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.545 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.545 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.545 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.545 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:25:43.545 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.545 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.545 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.545 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.545 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:25:43.545 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.545 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.545 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.546 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:25:43.898 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-01-23T08:25:43.898 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-01-23T08:25:43.992 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:25:44.004 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:25:44.004 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-01-23T08:25:44.004 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:25:44.004 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-01-23T08:25:44.004 INFO:teuthology.orchestra.run.smithi137.stdout: kernel x86_64 5.14.0-410.el9 baseos 6.0 M 2024-01-23T08:25:44.004 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-01-23T08:25:44.004 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core x86_64 5.14.0-410.el9 baseos 21 M 2024-01-23T08:25:44.004 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules x86_64 5.14.0-410.el9 baseos 39 M 2024-01-23T08:25:44.004 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-core x86_64 5.14.0-410.el9 baseos 33 M 2024-01-23T08:25:44.004 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:25:44.005 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-01-23T08:25:44.005 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:25:44.005 INFO:teuthology.orchestra.run.smithi137.stdout:Install 4 Packages 2024-01-23T08:25:44.005 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:25:44.005 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 99 M 2024-01-23T08:25:44.005 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 122 M 2024-01-23T08:25:44.006 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-01-23T08:25:44.401 INFO:teuthology.orchestra.run.smithi137.stdout:(1/4): kernel-5.14.0-410.el9.x86_64.rpm 21 MB/s | 6.0 MB 00:00 2024-01-23T08:25:44.743 INFO:teuthology.orchestra.run.smithi137.stdout:(2/4): kernel-core-5.14.0-410.el9.x86_64.rpm 33 MB/s | 21 MB 00:00 2024-01-23T08:25:45.276 INFO:teuthology.orchestra.run.smithi137.stdout:(3/4): kernel-modules-5.14.0-410.el9.x86_64.rpm 34 MB/s | 39 MB 00:01 2024-01-23T08:25:45.594 INFO:teuthology.orchestra.run.smithi137.stdout:(4/4): kernel-modules-core-5.14.0-410.el9.x86_6 28 MB/s | 33 MB 00:01 2024-01-23T08:25:45.594 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:25:45.594 INFO:teuthology.orchestra.run.smithi137.stdout:Total 62 MB/s | 99 MB 00:01 2024-01-23T08:25:46.481 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-01-23T08:25:46.854 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-01-23T08:25:46.854 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-01-23T08:25:47.155 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-01-23T08:25:48.526 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-01-23T08:25:48.527 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-01-23T08:25:49.412 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:25:49.412 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:25:49.412 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:25:49.412 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.412 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:25:49.412 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.412 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:25:49.412 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.412 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:25:49.412 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.413 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:25:49.413 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.413 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:25:49.413 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.413 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:25:49.413 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.413 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:25:49.413 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.413 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:25:49.413 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.413 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:25:49.413 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.413 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:25:49.413 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.413 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:25:49.414 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.414 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:25:49.414 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.414 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:25:49.414 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.414 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:25:49.414 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.414 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:25:49.414 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.414 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:25:49.414 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.414 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:25:49.414 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.414 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:25:49.414 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.415 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:25:49.415 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.415 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:25:49.415 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.415 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:25:49.415 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.415 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:25:49.415 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.415 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:25:49.416 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.416 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:25:49.416 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.416 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:25:49.416 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.416 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:25:49.416 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.416 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:25:49.416 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.416 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:25:49.416 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.416 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:25:49.416 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.416 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:25:49.416 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.417 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:25:49.417 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.417 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:25:49.417 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.417 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:25:49.417 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.417 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:25:49.417 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.417 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:25:49.417 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.417 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:25:49.417 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.417 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.417 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.417 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:25:49.417 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.417 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.418 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.418 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.418 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:25:49.418 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.418 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.418 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.418 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.418 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.418 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.418 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:25:49.418 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.418 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.418 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.418 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.418 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:25:49.418 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.418 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.419 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.419 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.419 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:25:49.419 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.419 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.419 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.419 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.419 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:25:49.419 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.419 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.419 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.419 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.419 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:25:49.419 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.419 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.419 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.420 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.420 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:25:49.420 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.420 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.420 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.420 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:25:49.787 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-01-23T08:25:49.787 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-01-23T08:25:49.894 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:25:49.906 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:25:49.906 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-01-23T08:25:49.906 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:25:49.906 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-23T08:25:49.906 INFO:teuthology.orchestra.run.smithi059.stdout: kernel x86_64 5.14.0-410.el9 baseos 6.0 M 2024-01-23T08:25:49.906 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-01-23T08:25:49.906 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-core x86_64 5.14.0-410.el9 baseos 21 M 2024-01-23T08:25:49.906 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules x86_64 5.14.0-410.el9 baseos 39 M 2024-01-23T08:25:49.906 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-core x86_64 5.14.0-410.el9 baseos 33 M 2024-01-23T08:25:49.906 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:25:49.906 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-23T08:25:49.906 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:25:49.906 INFO:teuthology.orchestra.run.smithi059.stdout:Install 4 Packages 2024-01-23T08:25:49.907 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:25:49.907 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 99 M 2024-01-23T08:25:49.907 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 122 M 2024-01-23T08:25:49.907 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-23T08:25:50.464 INFO:teuthology.orchestra.run.smithi059.stdout:(1/4): kernel-5.14.0-410.el9.x86_64.rpm 21 MB/s | 6.0 MB 00:00 2024-01-23T08:25:50.790 INFO:teuthology.orchestra.run.smithi059.stdout:(2/4): kernel-core-5.14.0-410.el9.x86_64.rpm 34 MB/s | 21 MB 00:00 2024-01-23T08:25:51.323 INFO:teuthology.orchestra.run.smithi059.stdout:(3/4): kernel-modules-5.14.0-410.el9.x86_64.rpm 34 MB/s | 39 MB 00:01 2024-01-23T08:25:51.396 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-01-23T08:25:51.690 INFO:teuthology.orchestra.run.smithi059.stdout:(4/4): kernel-modules-core-5.14.0-410.el9.x86_6 27 MB/s | 33 MB 00:01 2024-01-23T08:25:51.691 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:25:51.691 INFO:teuthology.orchestra.run.smithi059.stdout:Total 56 MB/s | 99 MB 00:01 2024-01-23T08:25:52.057 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-modules-core-5.14.0-410.el9.x86_64 1/4 2024-01-23T08:25:52.185 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-core-5.14.0-410.el9.x86_64 2/4 2024-01-23T08:25:52.611 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-23T08:25:53.066 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-23T08:25:53.066 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-23T08:25:54.452 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-23T08:25:54.453 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-23T08:25:54.943 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-5.14.0-410.el9.x86_64 2/4 2024-01-23T08:25:55.120 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-modules-5.14.0-410.el9.x86_64 3/4 2024-01-23T08:25:57.297 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-23T08:25:57.989 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-modules-core-5.14.0-410.el9.x86_64 1/4 2024-01-23T08:25:58.124 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-core-5.14.0-410.el9.x86_64 2/4 2024-01-23T08:25:59.810 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-5.14.0-410.el9.x86_64 3/4 2024-01-23T08:26:00.030 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-5.14.0-410.el9.x86_64 4/4 2024-01-23T08:26:01.059 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-core-5.14.0-410.el9.x86_64 2/4 2024-01-23T08:26:01.244 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-modules-5.14.0-410.el9.x86_64 3/4 2024-01-23T08:26:04.738 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-core-5.14.0-410.el9.x86_64 4/4 2024-01-23T08:26:06.312 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-5.14.0-410.el9.x86_64 3/4 2024-01-23T08:26:06.431 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-5.14.0-410.el9.x86_64 4/4 2024-01-23T08:26:11.527 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-core-5.14.0-410.el9.x86_64 4/4 2024-01-23T08:26:33.549 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-5.14.0-410.el9.x86_64 4/4 2024-01-23T08:26:33.550 INFO:teuthology.orchestra.run.smithi137.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-23T08:26:33.550 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:26:33.634 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-5.14.0-410.el9.x86_64 4/4 2024-01-23T08:26:34.119 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-5.14.0-410.el9.x86_64 4/4 2024-01-23T08:26:34.119 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-5.14.0-410.el9.x86_64 1/4 2024-01-23T08:26:34.119 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-core-5.14.0-410.el9.x86_64 2/4 2024-01-23T08:26:34.119 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-modules-5.14.0-410.el9.x86_64 3/4 2024-01-23T08:26:34.582 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-modules-core-5.14.0-410.el9.x86_64 4/4 2024-01-23T08:26:34.582 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:26:34.583 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-01-23T08:26:34.583 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-5.14.0-410.el9.x86_64 2024-01-23T08:26:34.583 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core-5.14.0-410.el9.x86_64 2024-01-23T08:26:34.583 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-5.14.0-410.el9.x86_64 2024-01-23T08:26:34.583 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-core-5.14.0-410.el9.x86_64 2024-01-23T08:26:34.583 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:26:34.583 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:26:35.092 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-01-23T08:26:35.244 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-5.14.0-410.el9.x86_64 2024-01-23T08:26:35.244 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-410.el9.x86_64 2024-01-23T08:26:35.244 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-410.el9.x86_64 2024-01-23T08:26:35.244 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-23T08:26:35.245 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-23T08:26:35.245 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-01-23T08:26:35.245 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi137.front.sepia.ceph.com, path=None, version=distro) 2024-01-23T08:26:35.245 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-01-23T08:26:35.366 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-5.14.0-410.el9.x86_64 2024-01-23T08:26:35.366 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-410.el9.x86_64 2024-01-23T08:26:35.367 DEBUG:teuthology.orchestra.run.smithi137:> sudo rpm -qi grub2-tools 2024-01-23T08:26:35.408 INFO:teuthology.orchestra.run.smithi137.stdout:Name : grub2-tools 2024-01-23T08:26:35.408 INFO:teuthology.orchestra.run.smithi137.stdout:Epoch : 1 2024-01-23T08:26:35.408 INFO:teuthology.orchestra.run.smithi137.stdout:Version : 2.06 2024-01-23T08:26:35.408 INFO:teuthology.orchestra.run.smithi137.stdout:Release : 68.el9 2024-01-23T08:26:35.409 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: x86_64 2024-01-23T08:26:35.409 INFO:teuthology.orchestra.run.smithi137.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-23T08:26:35.409 INFO:teuthology.orchestra.run.smithi137.stdout:Group : Unspecified 2024-01-23T08:26:35.409 INFO:teuthology.orchestra.run.smithi137.stdout:Size : 8264254 2024-01-23T08:26:35.409 INFO:teuthology.orchestra.run.smithi137.stdout:License : GPLv3+ 2024-01-23T08:26:35.409 INFO:teuthology.orchestra.run.smithi137.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-23T08:26:35.409 INFO:teuthology.orchestra.run.smithi137.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-23T08:26:35.409 INFO:teuthology.orchestra.run.smithi137.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-23T08:26:35.410 INFO:teuthology.orchestra.run.smithi137.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-23T08:26:35.410 INFO:teuthology.orchestra.run.smithi137.stdout:Packager : builder@centos.org 2024-01-23T08:26:35.410 INFO:teuthology.orchestra.run.smithi137.stdout:Vendor : CentOS 2024-01-23T08:26:35.410 INFO:teuthology.orchestra.run.smithi137.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-23T08:26:35.410 INFO:teuthology.orchestra.run.smithi137.stdout:Summary : Support tools for GRUB. 2024-01-23T08:26:35.410 INFO:teuthology.orchestra.run.smithi137.stdout:Description : 2024-01-23T08:26:35.410 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:26:35.410 INFO:teuthology.orchestra.run.smithi137.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-23T08:26:35.410 INFO:teuthology.orchestra.run.smithi137.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-23T08:26:35.410 INFO:teuthology.orchestra.run.smithi137.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-23T08:26:35.411 INFO:teuthology.orchestra.run.smithi137.stdout:hardware devices. 2024-01-23T08:26:35.411 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:26:35.411 INFO:teuthology.orchestra.run.smithi137.stdout:This subpackage provides tools for support of all platforms. 2024-01-23T08:26:35.412 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-23T08:26:35.412 INFO:teuthology.task.kernel:Updating grub on smithi137 to boot 5.14.0-410.el9.x86_64 2024-01-23T08:26:35.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-23T08:26:36.040 INFO:teuthology.orchestra.run.smithi137.stderr:Generating grub configuration file ... 2024-01-23T08:26:37.481 INFO:teuthology.orchestra.run.smithi137.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-23T08:26:37.513 INFO:teuthology.orchestra.run.smithi137.stderr:done 2024-01-23T08:26:37.515 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-01-23T08:26:37.530 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.KcjVLy2Yw2 2024-01-23T08:26:37.531 DEBUG:teuthology.orchestra.run.smithi137:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.KcjVLy2Yw2 2024-01-23T08:26:37.601 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.KcjVLy2Yw2 2024-01-23T08:26:37.742 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.KcjVLy2Yw2 is 6KB 2024-01-23T08:26:37.791 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.KcjVLy2Yw2 2024-01-23T08:26:37.805 DEBUG:teuthology.orchestra.run.smithi137:> sudo /bin/ls /boot/loader/entries || true 2024-01-23T08:26:37.877 INFO:teuthology.orchestra.run.smithi137.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-23T08:26:37.877 INFO:teuthology.orchestra.run.smithi137.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-23T08:26:37.877 INFO:teuthology.orchestra.run.smithi137.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-23T08:26:37.877 INFO:teuthology.orchestra.run.smithi137.stdout:15949e3c88704300882482eddfd7baae-5.14.0-410.el9.x86_64.conf 2024-01-23T08:26:37.878 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-410.el9.x86_64 2024-01-23T08:26:38.035 DEBUG:teuthology.orchestra.run.smithi137:> sudo shutdown -r now 2024-01-23T08:26:42.090 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-core-5.14.0-410.el9.x86_64 4/4 2024-01-23T08:26:42.091 INFO:teuthology.orchestra.run.smithi059.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-23T08:26:42.091 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:26:42.181 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-5.14.0-410.el9.x86_64 4/4 2024-01-23T08:26:42.580 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-5.14.0-410.el9.x86_64 4/4 2024-01-23T08:26:42.580 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-5.14.0-410.el9.x86_64 1/4 2024-01-23T08:26:42.580 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-core-5.14.0-410.el9.x86_64 2/4 2024-01-23T08:26:42.580 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-modules-5.14.0-410.el9.x86_64 3/4 2024-01-23T08:26:43.064 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-modules-core-5.14.0-410.el9.x86_64 4/4 2024-01-23T08:26:43.064 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:26:43.064 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-23T08:26:43.064 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-5.14.0-410.el9.x86_64 2024-01-23T08:26:43.064 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-core-5.14.0-410.el9.x86_64 2024-01-23T08:26:43.064 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-5.14.0-410.el9.x86_64 2024-01-23T08:26:43.064 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-core-5.14.0-410.el9.x86_64 2024-01-23T08:26:43.064 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:26:43.064 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:26:43.581 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q kernel | sort -rV | head -n 1 2024-01-23T08:26:43.724 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-5.14.0-410.el9.x86_64 2024-01-23T08:26:43.724 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-410.el9.x86_64 2024-01-23T08:26:43.724 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-410.el9.x86_64 2024-01-23T08:26:43.725 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-23T08:26:43.725 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-23T08:26:43.725 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-01-23T08:26:43.725 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi059.front.sepia.ceph.com, path=None, version=distro) 2024-01-23T08:26:43.725 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q kernel | sort -rV | head -n 1 2024-01-23T08:26:43.849 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-5.14.0-410.el9.x86_64 2024-01-23T08:26:43.850 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-410.el9.x86_64 2024-01-23T08:26:43.850 DEBUG:teuthology.orchestra.run.smithi059:> sudo rpm -qi grub2-tools 2024-01-23T08:26:43.897 INFO:teuthology.orchestra.run.smithi059.stdout:Name : grub2-tools 2024-01-23T08:26:43.897 INFO:teuthology.orchestra.run.smithi059.stdout:Epoch : 1 2024-01-23T08:26:43.897 INFO:teuthology.orchestra.run.smithi059.stdout:Version : 2.06 2024-01-23T08:26:43.897 INFO:teuthology.orchestra.run.smithi059.stdout:Release : 68.el9 2024-01-23T08:26:43.897 INFO:teuthology.orchestra.run.smithi059.stdout:Architecture: x86_64 2024-01-23T08:26:43.897 INFO:teuthology.orchestra.run.smithi059.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-23T08:26:43.898 INFO:teuthology.orchestra.run.smithi059.stdout:Group : Unspecified 2024-01-23T08:26:43.898 INFO:teuthology.orchestra.run.smithi059.stdout:Size : 8264254 2024-01-23T08:26:43.898 INFO:teuthology.orchestra.run.smithi059.stdout:License : GPLv3+ 2024-01-23T08:26:43.898 INFO:teuthology.orchestra.run.smithi059.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-23T08:26:43.898 INFO:teuthology.orchestra.run.smithi059.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-23T08:26:43.898 INFO:teuthology.orchestra.run.smithi059.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-23T08:26:43.898 INFO:teuthology.orchestra.run.smithi059.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-23T08:26:43.898 INFO:teuthology.orchestra.run.smithi059.stdout:Packager : builder@centos.org 2024-01-23T08:26:43.898 INFO:teuthology.orchestra.run.smithi059.stdout:Vendor : CentOS 2024-01-23T08:26:43.898 INFO:teuthology.orchestra.run.smithi059.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-23T08:26:43.898 INFO:teuthology.orchestra.run.smithi059.stdout:Summary : Support tools for GRUB. 2024-01-23T08:26:43.898 INFO:teuthology.orchestra.run.smithi059.stdout:Description : 2024-01-23T08:26:43.898 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:26:43.898 INFO:teuthology.orchestra.run.smithi059.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-23T08:26:43.899 INFO:teuthology.orchestra.run.smithi059.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-23T08:26:43.899 INFO:teuthology.orchestra.run.smithi059.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-23T08:26:43.899 INFO:teuthology.orchestra.run.smithi059.stdout:hardware devices. 2024-01-23T08:26:43.899 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:26:43.899 INFO:teuthology.orchestra.run.smithi059.stdout:This subpackage provides tools for support of all platforms. 2024-01-23T08:26:43.900 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-23T08:26:43.900 INFO:teuthology.task.kernel:Updating grub on smithi059 to boot 5.14.0-410.el9.x86_64 2024-01-23T08:26:43.900 DEBUG:teuthology.orchestra.run.smithi059:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-23T08:26:44.380 INFO:teuthology.orchestra.run.smithi059.stderr:Generating grub configuration file ... 2024-01-23T08:26:45.760 INFO:teuthology.orchestra.run.smithi059.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-23T08:26:45.774 INFO:teuthology.orchestra.run.smithi059.stderr:done 2024-01-23T08:26:45.776 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2024-01-23T08:26:45.791 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.OvZpP7XYUG 2024-01-23T08:26:45.792 DEBUG:teuthology.orchestra.run.smithi059:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.OvZpP7XYUG 2024-01-23T08:26:45.864 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.OvZpP7XYUG 2024-01-23T08:26:46.042 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.OvZpP7XYUG is 6KB 2024-01-23T08:26:46.091 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.OvZpP7XYUG 2024-01-23T08:26:46.105 DEBUG:teuthology.orchestra.run.smithi059:> sudo /bin/ls /boot/loader/entries || true 2024-01-23T08:26:46.181 INFO:teuthology.orchestra.run.smithi059.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-23T08:26:46.181 INFO:teuthology.orchestra.run.smithi059.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-23T08:26:46.181 INFO:teuthology.orchestra.run.smithi059.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-23T08:26:46.181 INFO:teuthology.orchestra.run.smithi059.stdout:15949e3c88704300882482eddfd7baae-5.14.0-410.el9.x86_64.conf 2024-01-23T08:26:46.182 DEBUG:teuthology.orchestra.run.smithi059:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-410.el9.x86_64 2024-01-23T08:26:46.341 DEBUG:teuthology.orchestra.run.smithi059:> sudo shutdown -r now 2024-01-23T08:26:46.444 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-01-23T08:27:08.046 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-01-23T08:27:08.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T08:27:08.048 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T08:27:16.348 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-01-23T08:27:16.348 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-01-23T08:27:16.349 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T08:27:42.524 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T08:27:51.532 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T08:27:51.533 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T08:28:09.920 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T08:28:16.356 DEBUG:teuthology.orchestra.remote:timed out 2024-01-23T08:28:21.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T08:28:21.929 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T08:28:22.367 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-01-23T08:28:22.806 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T08:28:22.806 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-410.el9.x86_64"... 2024-01-23T08:28:22.807 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-01-23T08:28:22.821 INFO:teuthology.orchestra.run.smithi137.stdout:5.14.0-410.el9.x86_64 2024-01-23T08:28:22.821 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-410.el9.x86_64 vs 5.14.0-410.el9.x86_64 2024-01-23T08:28:22.821 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-23T08:28:22.821 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-23T08:28:23.822 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-01-23T08:28:23.822 DEBUG:teuthology.orchestra.run.smithi137:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-23T08:28:23.907 INFO:teuthology.orchestra.run.smithi137.stdout:ttyS1 2024-01-23T08:28:23.956 DEBUG:teuthology.parallel:result is None 2024-01-23T08:28:25.357 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-01-23T08:28:25.358 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T08:28:25.975 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-01-23T08:28:26.410 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-01-23T08:28:26.410 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-410.el9.x86_64"... 2024-01-23T08:28:26.410 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2024-01-23T08:28:26.424 INFO:teuthology.orchestra.run.smithi059.stdout:5.14.0-410.el9.x86_64 2024-01-23T08:28:26.424 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-410.el9.x86_64 vs 5.14.0-410.el9.x86_64 2024-01-23T08:28:26.424 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-23T08:28:26.424 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-23T08:28:27.425 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-01-23T08:28:27.425 DEBUG:teuthology.orchestra.run.smithi059:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-23T08:28:27.520 INFO:teuthology.orchestra.run.smithi059.stdout:ttyS1 2024-01-23T08:28:27.562 DEBUG:teuthology.parallel:result is None 2024-01-23T08:28:27.563 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-23T08:28:27.569 INFO:teuthology.task.internal:Creating test directory... 2024-01-23T08:28:27.570 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-23T08:28:27.572 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-23T08:28:27.588 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-23T08:28:27.608 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-23T08:28:27.638 INFO:teuthology.task.internal:Creating archive directory... 2024-01-23T08:28:27.639 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-23T08:28:27.641 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-23T08:28:27.679 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-23T08:28:27.685 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-23T08:28:27.685 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-23T08:28:27.703 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-23T08:28:27.732 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-23T08:28:27.740 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-23T08:28:27.745 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-23T08:28:27.753 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-23T08:28:27.754 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-23T08:28:27.764 INFO:teuthology.task.internal:Configuring sudo... 2024-01-23T08:28:27.764 DEBUG:teuthology.orchestra.run.smithi059:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-23T08:28:27.789 DEBUG:teuthology.orchestra.run.smithi137:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-23T08:28:27.824 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-23T08:28:27.833 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-23T08:28:27.833 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-23T08:28:27.855 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-23T08:28:27.879 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-23T08:28:27.946 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-23T08:28:28.002 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-23T08:28:28.077 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-23T08:28:28.127 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-23T08:28:28.127 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-23T08:28:28.194 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-23T08:28:28.229 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-23T08:28:28.289 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-23T08:28:28.367 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-23T08:28:28.413 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-23T08:28:28.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-23T08:28:28.481 DEBUG:teuthology.orchestra.run.smithi059:> sudo service rsyslog restart 2024-01-23T08:28:28.485 DEBUG:teuthology.orchestra.run.smithi137:> sudo service rsyslog restart 2024-01-23T08:28:28.532 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-23T08:28:28.573 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-23T08:28:29.074 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-23T08:28:29.082 INFO:teuthology.task.internal:Starting timer... 2024-01-23T08:28:29.082 INFO:teuthology.run_tasks:Running task pcp... 2024-01-23T08:28:29.118 INFO:teuthology.run_tasks:Running task selinux... 2024-01-23T08:28:29.149 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-23T08:28:29.150 DEBUG:teuthology.orchestra.run.smithi059:> sudo service auditd rotate 2024-01-23T08:28:29.206 INFO:teuthology.orchestra.run.smithi059.stdout:Rotating logs: 2024-01-23T08:28:29.208 DEBUG:teuthology.orchestra.run.smithi137:> sudo service auditd rotate 2024-01-23T08:28:29.270 INFO:teuthology.orchestra.run.smithi137.stdout:Rotating logs: 2024-01-23T08:28:29.272 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-23T08:28:29.272 DEBUG:teuthology.orchestra.run.smithi059:> /usr/sbin/getenforce 2024-01-23T08:28:29.287 INFO:teuthology.orchestra.run.smithi059.stdout:Permissive 2024-01-23T08:28:29.288 DEBUG:teuthology.orchestra.run.smithi137:> /usr/sbin/getenforce 2024-01-23T08:28:29.318 INFO:teuthology.orchestra.run.smithi137.stdout:Permissive 2024-01-23T08:28:29.318 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi059.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi137.front.sepia.ceph.com': 'permissive'} 2024-01-23T08:28:29.318 DEBUG:teuthology.orchestra.run.smithi059:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-23T08:28:29.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-23T08:28:29.354 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-23T08:28:29.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-23T08:28:29.386 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-23T08:28:29.386 DEBUG:teuthology.orchestra.run.smithi059:> sudo /usr/sbin/setenforce permissive 2024-01-23T08:28:29.423 DEBUG:teuthology.orchestra.run.smithi137:> sudo /usr/sbin/setenforce permissive 2024-01-23T08:28:29.470 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-23T08:28:29.480 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-23T08:28:29.497 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-23T08:28:29.498 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi059.front.sepia.ceph.com,smithi137.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-23T08:32:39.240 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi059.front.sepia.ceph.com'), Remote(name='ubuntu@smithi137.front.sepia.ceph.com')] 2024-01-23T08:32:39.241 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-01-23T08:32:39.242 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T08:32:39.333 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-01-23T08:32:39.418 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-01-23T08:32:39.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T08:32:39.419 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T08:32:39.508 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-01-23T08:32:39.598 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T08:32:39.599 INFO:teuthology.run_tasks:Running task clock... 2024-01-23T08:32:39.609 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-23T08:32:39.609 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-23T08:32:39.610 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-23T08:32:39.612 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-23T08:32:39.613 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-23T08:32:39.641 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-23T08:32:39.658 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-23T08:32:39.669 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-23T08:32:39.686 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-23T08:32:39.687 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: ntpd: command not found 2024-01-23T08:32:39.703 INFO:teuthology.orchestra.run.smithi059.stdout:506 Cannot talk to daemon 2024-01-23T08:32:39.715 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: ntpd: command not found 2024-01-23T08:32:39.716 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-23T08:32:39.728 INFO:teuthology.orchestra.run.smithi137.stdout:506 Cannot talk to daemon 2024-01-23T08:32:39.733 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-23T08:32:39.745 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-23T08:32:39.761 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-23T08:32:39.785 INFO:teuthology.orchestra.run.smithi059.stderr:bash: line 1: ntpq: command not found 2024-01-23T08:32:39.789 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-23T08:32:39.790 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2024-01-23T08:32:39.790 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-23T08:32:39.790 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-23T08:32:39.790 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-23T08:32:39.790 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-23T08:32:39.812 INFO:teuthology.orchestra.run.smithi137.stderr:bash: line 1: ntpq: command not found 2024-01-23T08:32:39.816 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-23T08:32:39.816 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-01-23T08:32:39.816 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-23T08:32:39.816 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-23T08:32:39.816 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-23T08:32:39.816 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-23T08:32:39.816 INFO:teuthology.run_tasks:Running task install... 2024-01-23T08:32:39.824 DEBUG:teuthology.task.install:project ceph 2024-01-23T08:32:39.824 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d'}} 2024-01-23T08:32:39.824 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d'} 2024-01-23T08:32:39.824 INFO:teuthology.task.install:Using flavor: default 2024-01-23T08:32:39.831 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-23T08:32:39.831 INFO:teuthology.task.install:extra packages: [] 2024-01-23T08:32:39.831 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d', 'tag': None, 'wait_for_package': False} 2024-01-23T08:32:39.831 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-23T08:32:39.832 INFO:teuthology.packaging:ref: None 2024-01-23T08:32:39.832 INFO:teuthology.packaging:tag: None 2024-01-23T08:32:39.832 INFO:teuthology.packaging:branch: quincy 2024-01-23T08:32:39.832 INFO:teuthology.packaging:sha1: 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-23T08:32:39.832 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-01-23T08:32:39.834 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d', 'tag': None, 'wait_for_package': False} 2024-01-23T08:32:39.834 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-23T08:32:39.834 INFO:teuthology.packaging:ref: None 2024-01-23T08:32:39.834 INFO:teuthology.packaging:tag: None 2024-01-23T08:32:39.834 INFO:teuthology.packaging:branch: quincy 2024-01-23T08:32:39.834 INFO:teuthology.packaging:sha1: 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-23T08:32:39.834 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-01-23T08:32:40.032 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/b824e5befce1d427fda2bc90ae52665f0c7e2770/centos/9/flavors/default/ 2024-01-23T08:32:40.032 INFO:teuthology.task.install.rpm:Package version is 17.2.7-397.gb824e5be 2024-01-23T08:32:40.045 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/b824e5befce1d427fda2bc90ae52665f0c7e2770/centos/9/flavors/default/ 2024-01-23T08:32:40.045 INFO:teuthology.task.install.rpm:Package version is 17.2.7-397.gb824e5be 2024-01-23T08:32:40.146 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/b824e5befce1d427fda2bc90ae52665f0c7e2770/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/b824e5befce1d427fda2bc90ae52665f0c7e2770/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/b824e5befce1d427fda2bc90ae52665f0c7e2770/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-23T08:32:40.146 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-23T08:32:40.146 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-23T08:32:40.174 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-23T08:32:40.174 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-23T08:32:40.174 INFO:teuthology.packaging:ref: None 2024-01-23T08:32:40.174 INFO:teuthology.packaging:tag: None 2024-01-23T08:32:40.174 INFO:teuthology.packaging:branch: quincy 2024-01-23T08:32:40.174 INFO:teuthology.packaging:sha1: 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-23T08:32:40.174 DEBUG:teuthology.orchestra.run.smithi137:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-23T08:32:40.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-23T08:32:40.302 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/b824e5befce1d427fda2bc90ae52665f0c7e2770/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/b824e5befce1d427fda2bc90ae52665f0c7e2770/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/b824e5befce1d427fda2bc90ae52665f0c7e2770/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-23T08:32:40.303 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-23T08:32:40.303 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-23T08:32:40.322 DEBUG:teuthology.orchestra.run.smithi137:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-23T08:32:40.331 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-23T08:32:40.331 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-23T08:32:40.331 INFO:teuthology.packaging:ref: None 2024-01-23T08:32:40.331 INFO:teuthology.packaging:tag: None 2024-01-23T08:32:40.331 INFO:teuthology.packaging:branch: quincy 2024-01-23T08:32:40.331 INFO:teuthology.packaging:sha1: 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-23T08:32:40.331 DEBUG:teuthology.orchestra.run.smithi059:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-23T08:32:40.388 INFO:teuthology.orchestra.run.smithi137.stdout:check_obsoletes = 1 2024-01-23T08:32:40.390 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum clean all 2024-01-23T08:32:40.398 DEBUG:teuthology.orchestra.run.smithi059:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-23T08:32:40.478 DEBUG:teuthology.orchestra.run.smithi059:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-23T08:32:40.545 INFO:teuthology.orchestra.run.smithi059.stdout:check_obsoletes = 1 2024-01-23T08:32:40.546 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum clean all 2024-01-23T08:32:40.627 INFO:teuthology.orchestra.run.smithi137.stdout:73 files removed 2024-01-23T08:32:40.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-radosgw 2024-01-23T08:32:40.795 INFO:teuthology.orchestra.run.smithi059.stdout:73 files removed 2024-01-23T08:32:40.821 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-radosgw 2024-01-23T08:32:41.551 INFO:teuthology.orchestra.run.smithi137.stdout:ceph packages for x86_64 134 kB/s | 77 kB 00:00 2024-01-23T08:32:41.603 INFO:teuthology.orchestra.run.smithi059.stdout:ceph packages for x86_64 169 kB/s | 77 kB 00:00 2024-01-23T08:32:41.952 INFO:teuthology.orchestra.run.smithi137.stdout:ceph noarch packages 26 kB/s | 9.2 kB 00:00 2024-01-23T08:32:42.154 INFO:teuthology.orchestra.run.smithi059.stdout:ceph noarch packages 18 kB/s | 9.2 kB 00:00 2024-01-23T08:32:42.502 INFO:teuthology.orchestra.run.smithi137.stdout:ceph source packages 3.5 kB/s | 1.8 kB 00:00 2024-01-23T08:32:42.579 INFO:teuthology.orchestra.run.smithi059.stdout:ceph source packages 4.6 kB/s | 1.8 kB 00:00 2024-01-23T08:32:43.229 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS-9 - AppStream 11 MB/s | 7.8 MB 00:00 2024-01-23T08:32:43.298 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - AppStream 12 MB/s | 7.8 MB 00:00 2024-01-23T08:32:44.888 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.5 MB 00:00 2024-01-23T08:32:44.973 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.5 MB 00:00 2024-01-23T08:32:45.793 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-01-23T08:32:47.698 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-01-23T08:32:48.213 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - BaseOS 2.9 MB/s | 8.0 MB 00:02 2024-01-23T08:32:51.598 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - CRB 15 MB/s | 5.9 MB 00:00 2024-01-23T08:32:53.457 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 15 kB 00:00 2024-01-23T08:32:55.333 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2024-01-23T08:33:00.283 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-01-23T08:33:02.786 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:33:02.786 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:33:02.786 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:33:02.786 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.786 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:33:02.786 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.786 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:33:02.787 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.787 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:33:02.787 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.787 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:33:02.787 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.787 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:33:02.787 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.787 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:33:02.787 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.787 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:33:02.787 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.787 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:33:02.787 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.788 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:33:02.788 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.788 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:33:02.788 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.788 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:33:02.788 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.788 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:33:02.788 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.788 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:33:02.788 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.788 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:33:02.788 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.789 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:33:02.789 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.789 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:33:02.789 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.789 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:33:02.789 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.789 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:33:02.789 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.789 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:33:02.789 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.789 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:33:02.789 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.789 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:33:02.789 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.790 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:33:02.790 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.790 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:33:02.790 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.790 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:33:02.790 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.790 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:33:02.790 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.790 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:33:02.790 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.790 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:33:02.790 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.790 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:33:02.790 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.791 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:33:02.791 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.791 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:33:02.791 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.791 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:33:02.791 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.791 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:33:02.791 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.791 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:33:02.791 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.791 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:33:02.791 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.791 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:33:02.791 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.791 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:33:02.792 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.792 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.792 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.792 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:33:02.792 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.792 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.792 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.792 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.792 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:33:02.792 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.792 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.792 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.792 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.792 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.792 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.793 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:33:02.793 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.793 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.793 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.793 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.793 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:33:02.793 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.793 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.793 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.793 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.793 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:33:02.793 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.793 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.793 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.794 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.794 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:33:02.794 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.794 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.794 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.794 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.794 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:33:02.794 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.794 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.794 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.794 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.794 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:33:02.794 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.795 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.795 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:33:02.795 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:33:03.304 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:33:03.314 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:33:03.314 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-01-23T08:33:03.314 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:33:03.314 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-01-23T08:33:03.314 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw x86_64 2:17.2.7-397.gb824e5be.el9 ceph 14 M 2024-01-23T08:33:03.314 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrading: 2024-01-23T08:33:03.314 INFO:teuthology.orchestra.run.smithi137.stdout: librados2 x86_64 2:17.2.7-397.gb824e5be.el9 ceph 3.1 M 2024-01-23T08:33:03.314 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1 x86_64 2:17.2.7-397.gb824e5be.el9 ceph 2.9 M 2024-01-23T08:33:03.314 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-01-23T08:33:03.314 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base x86_64 2:17.2.7-397.gb824e5be.el9 ceph 5.0 M 2024-01-23T08:33:03.315 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common x86_64 2:17.2.7-397.gb824e5be.el9 ceph 23 M 2024-01-23T08:33:03.315 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux x86_64 2:17.2.7-397.gb824e5be.el9 ceph 25 k 2024-01-23T08:33:03.315 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-23T08:33:03.315 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2 x86_64 2:17.2.7-397.gb824e5be.el9 ceph 635 k 2024-01-23T08:33:03.315 INFO:teuthology.orchestra.run.smithi137.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-23T08:33:03.315 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-01-23T08:33:03.315 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-01-23T08:33:03.315 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1 x86_64 2:17.2.7-397.gb824e5be.el9 ceph 459 k 2024-01-23T08:33:03.315 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-01-23T08:33:03.315 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2 x86_64 2:17.2.7-397.gb824e5be.el9 ceph 4.9 M 2024-01-23T08:33:03.315 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-23T08:33:03.315 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-01-23T08:33:03.315 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse x86_64 2:17.2.7-397.gb824e5be.el9 ceph 45 k 2024-01-23T08:33:03.315 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common x86_64 2:17.2.7-397.gb824e5be.el9 ceph 116 k 2024-01-23T08:33:03.315 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs x86_64 2:17.2.7-397.gb824e5be.el9 ceph 156 k 2024-01-23T08:33:03.316 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-01-23T08:33:03.316 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados x86_64 2:17.2.7-397.gb824e5be.el9 ceph 320 k 2024-01-23T08:33:03.316 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd x86_64 2:17.2.7-397.gb824e5be.el9 ceph 293 k 2024-01-23T08:33:03.316 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw x86_64 2:17.2.7-397.gb824e5be.el9 ceph 99 k 2024-01-23T08:33:03.316 INFO:teuthology.orchestra.run.smithi137.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-23T08:33:03.316 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:03.316 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-01-23T08:33:03.316 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:33:03.316 INFO:teuthology.orchestra.run.smithi137.stdout:Install 22 Packages 2024-01-23T08:33:03.316 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrade 2 Packages 2024-01-23T08:33:03.316 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:03.317 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 58 M 2024-01-23T08:33:03.317 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-01-23T08:33:04.713 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-01-23T08:33:04.767 INFO:teuthology.orchestra.run.smithi137.stdout:(1/24): ceph-base-17.2.7-397.gb824e5be.el9.x86_ 4.1 MB/s | 5.0 MB 00:01 2024-01-23T08:33:04.976 INFO:teuthology.orchestra.run.smithi137.stdout:(2/24): ceph-radosgw-17.2.7-397.gb824e5be.el9.x 9.9 MB/s | 14 MB 00:01 2024-01-23T08:33:05.001 INFO:teuthology.orchestra.run.smithi137.stdout:(3/24): ceph-selinux-17.2.7-397.gb824e5be.el9.x 108 kB/s | 25 kB 00:00 2024-01-23T08:33:05.051 INFO:teuthology.orchestra.run.smithi137.stdout:(4/24): libcephfs2-17.2.7-397.gb824e5be.el9.x86 8.2 MB/s | 635 kB 00:00 2024-01-23T08:33:05.079 INFO:teuthology.orchestra.run.smithi137.stdout:(5/24): libradosstriper1-17.2.7-397.gb824e5be.e 5.9 MB/s | 459 kB 00:00 2024-01-23T08:33:05.169 INFO:teuthology.orchestra.run.smithi137.stdout:(6/24): python3-ceph-argparse-17.2.7-397.gb824e 491 kB/s | 45 kB 00:00 2024-01-23T08:33:05.261 INFO:teuthology.orchestra.run.smithi137.stdout:(7/24): python3-ceph-common-17.2.7-397.gb824e5b 1.2 MB/s | 116 kB 00:00 2024-01-23T08:33:05.352 INFO:teuthology.orchestra.run.smithi137.stdout:(8/24): librgw2-17.2.7-397.gb824e5be.el9.x86_64 16 MB/s | 4.9 MB 00:00 2024-01-23T08:33:05.378 INFO:teuthology.orchestra.run.smithi137.stdout:(9/24): python3-cephfs-17.2.7-397.gb824e5be.el9 1.3 MB/s | 156 kB 00:00 2024-01-23T08:33:05.420 INFO:teuthology.orchestra.run.smithi137.stdout:(10/24): python3-rados-17.2.7-397.gb824e5be.el9 4.7 MB/s | 320 kB 00:00 2024-01-23T08:33:05.453 INFO:teuthology.orchestra.run.smithi137.stdout:(11/24): python3-rbd-17.2.7-397.gb824e5be.el9.x 3.8 MB/s | 293 kB 00:00 2024-01-23T08:33:05.487 INFO:teuthology.orchestra.run.smithi137.stdout:(12/24): python3-rgw-17.2.7-397.gb824e5be.el9.x 1.5 MB/s | 99 kB 00:00 2024-01-23T08:33:05.829 INFO:teuthology.orchestra.run.smithi137.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 132 kB/s | 45 kB 00:00 2024-01-23T08:33:05.954 INFO:teuthology.orchestra.run.smithi137.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 320 kB/s | 160 kB 00:00 2024-01-23T08:33:06.146 INFO:teuthology.orchestra.run.smithi137.stdout:(15/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.5 MB/s | 292 kB 00:00 2024-01-23T08:33:06.288 INFO:teuthology.orchestra.run.smithi137.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 298 kB/s | 42 kB 00:00 2024-01-23T08:33:06.313 INFO:teuthology.orchestra.run.smithi137.stdout:(17/24): librdkafka-1.6.1-102.el9.x86_64.rpm 1.3 MB/s | 662 kB 00:00 2024-01-23T08:33:06.530 INFO:teuthology.orchestra.run.smithi137.stdout:(18/24): ceph-common-17.2.7-397.gb824e5be.el9.x 7.6 MB/s | 23 MB 00:02 2024-01-23T08:33:06.997 INFO:teuthology.orchestra.run.smithi137.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 144 kB/s | 67 kB 00:00 2024-01-23T08:33:07.056 INFO:teuthology.orchestra.run.smithi137.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 65 kB/s | 49 kB 00:00 2024-01-23T08:33:07.173 INFO:teuthology.orchestra.run.smithi137.stdout:(21/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 349 kB/s | 309 kB 00:00 2024-01-23T08:33:07.465 INFO:teuthology.orchestra.run.smithi137.stdout:(22/24): librbd1-17.2.7-397.gb824e5be.el9.x86_6 10 MB/s | 2.9 MB 00:00 2024-01-23T08:33:07.656 INFO:teuthology.orchestra.run.smithi137.stdout:(23/24): librados2-17.2.7-397.gb824e5be.el9.x86 5.1 MB/s | 3.1 MB 00:00 2024-01-23T08:33:07.698 INFO:teuthology.orchestra.run.smithi137.stdout:(24/24): thrift-0.15.0-2.el9.x86_64.rpm 2.3 MB/s | 1.6 MB 00:00 2024-01-23T08:33:07.701 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:33:07.701 INFO:teuthology.orchestra.run.smithi137.stdout:Total 13 MB/s | 58 MB 00:04 2024-01-23T08:33:07.764 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-01-23T08:33:07.796 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-01-23T08:33:07.796 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-01-23T08:33:08.270 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-01-23T08:33:08.270 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-01-23T08:33:09.499 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-01-23T08:33:09.827 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-01-23T08:33:09.875 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librados2-2:17.2.7-397.gb824e5be.el9.x86_64 2/26 2024-01-23T08:33:09.978 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:17.2.7-397.gb824e5be.el9.x86_64 2/26 2024-01-23T08:33:10.054 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rados-2:17.2.7-397.gb824e5be.el9.x86_64 3/26 2024-01-23T08:33:10.077 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs2-2:17.2.7-397.gb824e5be.el9.x86_64 4/26 2024-01-23T08:33:10.181 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs2-2:17.2.7-397.gb824e5be.el9.x86_64 4/26 2024-01-23T08:33:10.236 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-01-23T08:33:10.265 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-01-23T08:33:10.708 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-01-23T08:33:10.738 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librgw2-2:17.2.7-397.gb824e5be.el9.x86_64 8/26 2024-01-23T08:33:10.784 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librgw2-2:17.2.7-397.gb824e5be.el9.x86_64 8/26 2024-01-23T08:33:10.797 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 5.9 MB 00:02 2024-01-23T08:33:10.812 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-01-23T08:33:10.854 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-argparse-2:17.2.7-397.gb824e5be.el9.x 10/26 2024-01-23T08:33:10.892 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cephfs-2:17.2.7-397.gb824e5be.el9.x86_64 11/26 2024-01-23T08:33:10.961 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rgw-2:17.2.7-397.gb824e5be.el9.x86_64 12/26 2024-01-23T08:33:10.990 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libradosstriper1-2:17.2.7-397.gb824e5be.el9.x86_64 13/26 2024-01-23T08:33:11.039 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libradosstriper1-2:17.2.7-397.gb824e5be.el9.x86_64 13/26 2024-01-23T08:33:11.078 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-01-23T08:33:11.124 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-01-23T08:33:11.156 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-01-23T08:33:11.459 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-01-23T08:33:11.486 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librbd1-2:17.2.7-397.gb824e5be.el9.x86_64 18/26 2024-01-23T08:33:11.575 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:17.2.7-397.gb824e5be.el9.x86_64 18/26 2024-01-23T08:33:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rbd-2:17.2.7-397.gb824e5be.el9.x86_64 19/26 2024-01-23T08:33:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-common-2:17.2.7-397.gb824e5be.el9.x86 20/26 2024-01-23T08:33:12.431 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 15 kB 00:00 2024-01-23T08:33:14.082 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:17.2.7-397.gb824e5be.el9.x86_64 21/26 2024-01-23T08:33:14.118 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-common-2:17.2.7-397.gb824e5be.el9.x86_64 21/26 2024-01-23T08:33:14.669 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:17.2.7-397.gb824e5be.el9.x86_64 21/26 2024-01-23T08:33:14.705 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-base-2:17.2.7-397.gb824e5be.el9.x86_64 22/26 2024-01-23T08:33:14.808 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-base-2:17.2.7-397.gb824e5be.el9.x86_64 22/26 2024-01-23T08:33:14.808 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-23T08:33:14.808 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-23T08:33:14.808 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:14.839 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-selinux-2:17.2.7-397.gb824e5be.el9.x86_64 23/26 2024-01-23T08:33:24.424 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-selinux-2:17.2.7-397.gb824e5be.el9.x86_64 23/26 2024-01-23T08:33:24.425 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /sys 2024-01-23T08:33:24.425 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /proc 2024-01-23T08:33:24.425 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /mnt 2024-01-23T08:33:24.425 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /var/tmp 2024-01-23T08:33:24.425 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /home 2024-01-23T08:33:24.425 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /root 2024-01-23T08:33:24.425 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /tmp 2024-01-23T08:33:24.425 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:25.731 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-radosgw-2:17.2.7-397.gb824e5be.el9.x86_64 24/26 2024-01-23T08:33:25.809 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-397.gb824e5be.el9.x86_64 24/26 2024-01-23T08:33:25.809 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-23T08:33:25.809 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-23T08:33:25.809 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-23T08:33:25.809 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-23T08:33:25.809 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:25.811 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-01-23T08:33:25.911 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-01-23T08:33:25.911 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-01-23T08:33:27.110 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-01-23T08:33:27.110 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-base-2:17.2.7-397.gb824e5be.el9.x86_64 1/26 2024-01-23T08:33:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-common-2:17.2.7-397.gb824e5be.el9.x86_64 2/26 2024-01-23T08:33:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-radosgw-2:17.2.7-397.gb824e5be.el9.x86_64 3/26 2024-01-23T08:33:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-selinux-2:17.2.7-397.gb824e5be.el9.x86_64 4/26 2024-01-23T08:33:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs2-2:17.2.7-397.gb824e5be.el9.x86_64 5/26 2024-01-23T08:33:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libradosstriper1-2:17.2.7-397.gb824e5be.el9.x86_64 6/26 2024-01-23T08:33:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librgw2-2:17.2.7-397.gb824e5be.el9.x86_64 7/26 2024-01-23T08:33:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-argparse-2:17.2.7-397.gb824e5be.el9.x 8/26 2024-01-23T08:33:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-common-2:17.2.7-397.gb824e5be.el9.x86 9/26 2024-01-23T08:33:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cephfs-2:17.2.7-397.gb824e5be.el9.x86_64 10/26 2024-01-23T08:33:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rados-2:17.2.7-397.gb824e5be.el9.x86_64 11/26 2024-01-23T08:33:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rbd-2:17.2.7-397.gb824e5be.el9.x86_64 12/26 2024-01-23T08:33:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rgw-2:17.2.7-397.gb824e5be.el9.x86_64 13/26 2024-01-23T08:33:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-01-23T08:33:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-01-23T08:33:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-01-23T08:33:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-01-23T08:33:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-01-23T08:33:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-01-23T08:33:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-01-23T08:33:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-01-23T08:33:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-01-23T08:33:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:17.2.7-397.gb824e5be.el9.x86_64 23/26 2024-01-23T08:33:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-01-23T08:33:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:17.2.7-397.gb824e5be.el9.x86_64 25/26 2024-01-23T08:33:28.019 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-01-23T08:33:28.020 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:28.020 INFO:teuthology.orchestra.run.smithi137.stdout:Upgraded: 2024-01-23T08:33:28.020 INFO:teuthology.orchestra.run.smithi137.stdout: librados2-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:33:28.020 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:33:28.020 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-01-23T08:33:28.020 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:33:28.020 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:33:28.020 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:33:28.020 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:33:28.020 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-23T08:33:28.020 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:33:28.020 INFO:teuthology.orchestra.run.smithi137.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-23T08:33:28.020 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-23T08:33:28.021 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-23T08:33:28.021 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:33:28.021 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-23T08:33:28.021 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:33:28.021 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-23T08:33:28.021 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-23T08:33:28.021 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:33:28.021 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:33:28.021 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:33:28.021 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-23T08:33:28.021 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:33:28.021 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:33:28.021 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:33:28.021 INFO:teuthology.orchestra.run.smithi137.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-23T08:33:28.021 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:28.021 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:33:28.353 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-test 2024-01-23T08:33:28.957 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:28 ago on Tue 23 Jan 2024 08:33:00 AM UTC. 2024-01-23T08:33:29.249 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:33:29.249 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:33:29.249 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:33:29.249 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.249 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:33:29.249 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.250 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:33:29.250 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.250 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:33:29.250 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.250 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:33:29.250 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.250 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:33:29.250 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.250 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:33:29.250 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.250 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:33:29.250 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.250 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:33:29.250 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.251 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:33:29.251 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.251 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:33:29.251 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.251 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:33:29.251 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.251 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:33:29.251 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.251 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:33:29.251 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.251 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:33:29.251 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.251 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:33:29.251 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.252 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:33:29.252 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.252 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:33:29.252 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.252 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:33:29.252 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.252 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:33:29.252 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.252 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:33:29.252 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.252 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:33:29.252 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.252 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:33:29.252 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.253 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:33:29.253 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.253 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:33:29.253 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.253 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:33:29.253 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.253 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:33:29.253 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.253 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:33:29.253 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.253 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:33:29.253 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.253 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:33:29.253 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.254 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:33:29.254 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.254 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:33:29.254 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.254 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:33:29.254 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.254 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:33:29.254 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.254 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:33:29.254 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.254 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:33:29.254 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.254 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:33:29.254 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.254 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.255 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.255 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:33:29.255 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.255 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.255 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.255 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.255 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:33:29.255 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.255 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.255 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.255 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.255 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.255 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.255 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:33:29.255 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.256 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.256 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.256 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.256 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:33:29.256 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.256 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.256 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.256 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.256 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:33:29.256 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.256 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.256 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.256 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.257 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:33:29.257 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.257 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.257 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.257 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.257 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:33:29.257 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.257 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.257 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.257 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.257 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:33:29.257 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.257 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.257 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.258 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:33:29.316 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:33:29.316 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:33:29.316 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-01-23T08:33:29.316 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:33:29.317 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-01-23T08:33:29.317 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test x86_64 2:17.2.7-397.gb824e5be.el9 ceph 48 M 2024-01-23T08:33:29.317 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-01-23T08:33:29.317 INFO:teuthology.orchestra.run.smithi137.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-01-23T08:33:29.317 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite x86_64 2:17.2.7-397.gb824e5be.el9 ceph 162 k 2024-01-23T08:33:29.317 INFO:teuthology.orchestra.run.smithi137.stdout: oniguruma x86_64 6.9.6-1.el9.5 baseos 218 k 2024-01-23T08:33:29.317 INFO:teuthology.orchestra.run.smithi137.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-01-23T08:33:29.317 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-01-23T08:33:29.317 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:29.317 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-01-23T08:33:29.317 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:33:29.317 INFO:teuthology.orchestra.run.smithi137.stdout:Install 6 Packages 2024-01-23T08:33:29.317 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:29.318 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 49 M 2024-01-23T08:33:29.318 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 195 M 2024-01-23T08:33:29.318 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-01-23T08:33:29.848 INFO:teuthology.orchestra.run.smithi137.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-01-23T08:33:29.906 INFO:teuthology.orchestra.run.smithi137.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.7 MB/s | 218 kB 00:00 2024-01-23T08:33:29.948 INFO:teuthology.orchestra.run.smithi137.stdout:(3/6): libcephsqlite-17.2.7-397.gb824e5be.el9.x 617 kB/s | 162 kB 00:00 2024-01-23T08:33:30.298 INFO:teuthology.orchestra.run.smithi137.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 182 kB/s | 64 kB 00:00 2024-01-23T08:33:30.490 INFO:teuthology.orchestra.run.smithi137.stdout:(5/6): socat-1.7.4.1-5.el9.x86_64.rpm 522 kB/s | 305 kB 00:00 2024-01-23T08:33:32.016 INFO:teuthology.orchestra.run.smithi137.stdout:(6/6): ceph-test-17.2.7-397.gb824e5be.el9.x86_6 21 MB/s | 48 MB 00:02 2024-01-23T08:33:32.018 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:33:32.018 INFO:teuthology.orchestra.run.smithi137.stdout:Total 18 MB/s | 49 MB 00:02 2024-01-23T08:33:32.068 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-01-23T08:33:32.087 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-01-23T08:33:32.087 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-01-23T08:33:32.487 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-01-23T08:33:32.487 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-01-23T08:33:33.073 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-01-23T08:33:33.167 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-01-23T08:33:33.208 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-01-23T08:33:33.248 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-01-23T08:33:33.290 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-01-23T08:33:33.318 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephsqlite-2:17.2.7-397.gb824e5be.el9.x86_64 5/6 2024-01-23T08:33:37.712 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephsqlite-2:17.2.7-397.gb824e5be.el9.x86_64 5/6 2024-01-23T08:33:37.784 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-test-2:17.2.7-397.gb824e5be.el9.x86_64 6/6 2024-01-23T08:33:40.025 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-test-2:17.2.7-397.gb824e5be.el9.x86_64 6/6 2024-01-23T08:33:40.025 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-test-2:17.2.7-397.gb824e5be.el9.x86_64 1/6 2024-01-23T08:33:40.025 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephsqlite-2:17.2.7-397.gb824e5be.el9.x86_64 2/6 2024-01-23T08:33:40.026 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2024-01-23T08:33:40.026 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2024-01-23T08:33:40.026 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2024-01-23T08:33:40.414 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2024-01-23T08:33:40.414 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:40.414 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-01-23T08:33:40.415 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:33:40.415 INFO:teuthology.orchestra.run.smithi137.stdout: jq-1.6-16.el9.x86_64 2024-01-23T08:33:40.415 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:33:40.415 INFO:teuthology.orchestra.run.smithi137.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-23T08:33:40.415 INFO:teuthology.orchestra.run.smithi137.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-23T08:33:40.415 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-23T08:33:40.415 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:40.415 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:33:40.700 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph 2024-01-23T08:33:41.311 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:41 ago on Tue 23 Jan 2024 08:33:00 AM UTC. 2024-01-23T08:33:41.615 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:33:41.615 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:33:41.615 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:33:41.615 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.615 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:33:41.615 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.615 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:33:41.615 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.615 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:33:41.616 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.616 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:33:41.616 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.616 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:33:41.616 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.616 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:33:41.616 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.616 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:33:41.616 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.616 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:33:41.616 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.617 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:33:41.617 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.617 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:33:41.617 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.617 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:33:41.617 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.617 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:33:41.617 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.617 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:33:41.617 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.617 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:33:41.618 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.618 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:33:41.618 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.618 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:33:41.618 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.618 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:33:41.618 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.618 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:33:41.618 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.618 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:33:41.618 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.619 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:33:41.619 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.619 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:33:41.619 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.619 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:33:41.619 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.619 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:33:41.619 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.619 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:33:41.619 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.619 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:33:41.619 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.619 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:33:41.620 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.620 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:33:41.620 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.620 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:33:41.620 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.620 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:33:41.620 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.620 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:33:41.620 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.620 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:33:41.620 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.620 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:33:41.621 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.621 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:33:41.621 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.621 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:33:41.621 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.621 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:33:41.621 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.621 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:33:41.621 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.621 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.621 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.621 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:33:41.622 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.622 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.622 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.622 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.622 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:33:41.622 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.622 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.622 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.622 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.622 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.622 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.622 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:33:41.622 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.623 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.623 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.623 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.623 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:33:41.623 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.623 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.623 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.623 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.623 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:33:41.623 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.623 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.623 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.624 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.624 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:33:41.624 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.624 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.624 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.624 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.624 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:33:41.624 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.624 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.624 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.624 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.624 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:33:41.625 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.625 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.625 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.625 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:33:41.686 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:33:41.688 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:33:41.688 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-01-23T08:33:41.688 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:33:41.688 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-01-23T08:33:41.689 INFO:teuthology.orchestra.run.smithi137.stdout: ceph x86_64 2:17.2.7-397.gb824e5be.el9 ceph 6.5 k 2024-01-23T08:33:41.689 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-01-23T08:33:41.689 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds x86_64 2:17.2.7-397.gb824e5be.el9 ceph 2.1 M 2024-01-23T08:33:41.689 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr x86_64 2:17.2.7-397.gb824e5be.el9 ceph 1.4 M 2024-01-23T08:33:41.689 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core noarch 2:17.2.7-397.gb824e5be.el9 ceph-noarch 239 k 2024-01-23T08:33:41.689 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon x86_64 2:17.2.7-397.gb824e5be.el9 ceph 3.7 M 2024-01-23T08:33:41.689 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd x86_64 2:17.2.7-397.gb824e5be.el9 ceph 15 M 2024-01-23T08:33:41.689 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-23T08:33:41.689 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-01-23T08:33:41.689 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-23T08:33:41.689 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-23T08:33:41.689 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-01-23T08:33:41.690 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-23T08:33:41.690 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-23T08:33:41.690 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-23T08:33:41.690 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-23T08:33:41.690 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-23T08:33:41.690 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-23T08:33:41.690 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-01-23T08:33:41.690 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-01-23T08:33:41.690 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-23T08:33:41.690 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-23T08:33:41.690 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-01-23T08:33:41.690 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-23T08:33:41.690 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-23T08:33:41.691 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-01-23T08:33:41.691 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-01-23T08:33:41.691 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests noarch 2.25.1-7.el9 baseos 124 k 2024-01-23T08:33:41.691 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-23T08:33:41.691 INFO:teuthology.orchestra.run.smithi137.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-01-23T08:33:41.691 INFO:teuthology.orchestra.run.smithi137.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-01-23T08:33:41.691 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-23T08:33:41.691 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-23T08:33:41.691 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-23T08:33:41.691 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:41.691 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-01-23T08:33:41.691 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:33:41.691 INFO:teuthology.orchestra.run.smithi137.stdout:Install 33 Packages 2024-01-23T08:33:41.692 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:41.692 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 27 M 2024-01-23T08:33:41.692 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 99 M 2024-01-23T08:33:41.692 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-01-23T08:33:42.259 INFO:teuthology.orchestra.run.smithi137.stdout:(1/33): ceph-17.2.7-397.gb824e5be.el9.x86_64.rp 38 kB/s | 6.5 kB 00:00 2024-01-23T08:33:42.459 INFO:teuthology.orchestra.run.smithi137.stdout:(2/33): ceph-mgr-17.2.7-397.gb824e5be.el9.x86_6 3.8 MB/s | 1.4 MB 00:00 2024-01-23T08:33:42.818 INFO:teuthology.orchestra.run.smithi137.stdout:(3/33): ceph-mds-17.2.7-397.gb824e5be.el9.x86_6 2.8 MB/s | 2.1 MB 00:00 2024-01-23T08:33:42.926 INFO:teuthology.orchestra.run.smithi137.stdout:(4/33): ceph-mgr-modules-core-17.2.7-397.gb824e 2.2 MB/s | 239 kB 00:00 2024-01-23T08:33:43.051 INFO:teuthology.orchestra.run.smithi137.stdout:(5/33): ceph-mon-17.2.7-397.gb824e5be.el9.x86_6 4.7 MB/s | 3.7 MB 00:00 2024-01-23T08:33:43.110 INFO:teuthology.orchestra.run.smithi137.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-01-23T08:33:43.319 INFO:teuthology.orchestra.run.smithi137.stdout:(7/33): ceph-osd-17.2.7-397.gb824e5be.el9.x86_6 18 MB/s | 15 MB 00:00 2024-01-23T08:33:43.360 INFO:teuthology.orchestra.run.smithi137.stdout:(8/33): python3-cryptography-36.0.1-4.el9.x86_6 4.0 MB/s | 1.2 MB 00:00 2024-01-23T08:33:43.411 INFO:teuthology.orchestra.run.smithi137.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 356 kB/s | 106 kB 00:00 2024-01-23T08:33:43.494 INFO:teuthology.orchestra.run.smithi137.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 771 kB/s | 135 kB 00:00 2024-01-23T08:33:43.536 INFO:teuthology.orchestra.run.smithi137.stdout:(11/33): python3-requests-2.25.1-7.el9.noarch.r 819 kB/s | 124 kB 00:00 2024-01-23T08:33:43.578 INFO:teuthology.orchestra.run.smithi137.stdout:(12/33): python3-urllib3-1.26.5-5.el9.noarch.rp 1.5 MB/s | 215 kB 00:00 2024-01-23T08:33:43.654 INFO:teuthology.orchestra.run.smithi137.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 297 kB/s | 35 kB 00:00 2024-01-23T08:33:43.679 INFO:teuthology.orchestra.run.smithi137.stdout:(14/33): python3-mako-1.1.4-6.el9.noarch.rpm 941 kB/s | 172 kB 00:00 2024-01-23T08:33:43.704 INFO:teuthology.orchestra.run.smithi137.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 407 kB/s | 51 kB 00:00 2024-01-23T08:33:43.729 INFO:teuthology.orchestra.run.smithi137.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 552 kB/s | 42 kB 00:00 2024-01-23T08:33:44.113 INFO:teuthology.orchestra.run.smithi137.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 100 kB/s | 43 kB 00:00 2024-01-23T08:33:44.230 INFO:teuthology.orchestra.run.smithi137.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 92 kB/s | 11 kB 00:00 2024-01-23T08:33:44.314 INFO:teuthology.orchestra.run.smithi137.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 282 kB/s | 172 kB 00:00 2024-01-23T08:33:44.347 INFO:teuthology.orchestra.run.smithi137.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 151 kB/s | 18 kB 00:00 2024-01-23T08:33:44.406 INFO:teuthology.orchestra.run.smithi137.stdout:(21/33): python3-cherrypy-18.6.1-2.el9.noarch.r 530 kB/s | 358 kB 00:00 2024-01-23T08:33:44.431 INFO:teuthology.orchestra.run.smithi137.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 197 kB/s | 23 kB 00:00 2024-01-23T08:33:44.473 INFO:teuthology.orchestra.run.smithi137.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 155 kB/s | 19 kB 00:00 2024-01-23T08:33:44.506 INFO:teuthology.orchestra.run.smithi137.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-01-23T08:33:44.557 INFO:teuthology.orchestra.run.smithi137.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 369 kB/s | 46 kB 00:00 2024-01-23T08:33:44.657 INFO:teuthology.orchestra.run.smithi137.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-01-23T08:33:44.674 INFO:teuthology.orchestra.run.smithi137.stdout:(27/33): python3-more-itertools-8.12.0-2.el9.no 393 kB/s | 79 kB 00:00 2024-01-23T08:33:44.699 INFO:teuthology.orchestra.run.smithi137.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 115 kB/s | 16 kB 00:00 2024-01-23T08:33:44.774 INFO:teuthology.orchestra.run.smithi137.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 768 kB/s | 90 kB 00:00 2024-01-23T08:33:44.825 INFO:teuthology.orchestra.run.smithi137.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 285 kB/s | 36 kB 00:00 2024-01-23T08:33:44.884 INFO:teuthology.orchestra.run.smithi137.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-01-23T08:33:44.950 INFO:teuthology.orchestra.run.smithi137.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 159 kB/s | 20 kB 00:00 2024-01-23T08:33:44.975 INFO:teuthology.orchestra.run.smithi137.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.1 MB/s | 427 kB 00:00 2024-01-23T08:33:44.981 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:33:44.981 INFO:teuthology.orchestra.run.smithi137.stdout:Total 8.2 MB/s | 27 MB 00:03 2024-01-23T08:33:45.097 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-01-23T08:33:45.133 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-01-23T08:33:45.134 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-01-23T08:33:45.505 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-01-23T08:33:45.505 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-01-23T08:33:46.740 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-01-23T08:33:46.795 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-01-23T08:33:46.827 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-01-23T08:33:46.859 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-01-23T08:33:46.951 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-01-23T08:33:47.020 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-01-23T08:33:47.062 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-01-23T08:33:47.097 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-01-23T08:33:47.127 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-01-23T08:33:47.161 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-01-23T08:33:47.195 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-01-23T08:33:47.235 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-01-23T08:33:47.287 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-01-23T08:33:47.321 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-01-23T08:33:47.363 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-01-23T08:33:47.439 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-01-23T08:33:47.576 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-01-23T08:33:47.662 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-01-23T08:33:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-01-23T08:33:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 19/33 2024-01-23T08:33:47.803 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-01-23T08:33:47.866 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-01-23T08:33:48.062 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-01-23T08:33:48.109 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-01-23T08:33:48.176 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-01-23T08:33:48.266 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-01-23T08:33:48.307 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-01-23T08:33:48.421 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-01-23T08:33:48.569 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-397.gb824e5be.el9.n 28/33 2024-01-23T08:33:48.601 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-2:17.2.7-397.gb824e5be.el9.x86_64 29/33 2024-01-23T08:33:48.627 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-2:17.2.7-397.gb824e5be.el9.x86_64 29/33 2024-01-23T08:33:48.627 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-23T08:33:48.627 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-23T08:33:48.627 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-23T08:33:48.627 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-23T08:33:48.627 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:49.960 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-osd-2:17.2.7-397.gb824e5be.el9.x86_64 30/33 2024-01-23T08:33:49.987 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-osd-2:17.2.7-397.gb824e5be.el9.x86_64 30/33 2024-01-23T08:33:49.987 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-23T08:33:49.987 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-23T08:33:49.987 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-23T08:33:49.987 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-23T08:33:49.987 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:50.369 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mon-2:17.2.7-397.gb824e5be.el9.x86_64 31/33 2024-01-23T08:33:50.392 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mon-2:17.2.7-397.gb824e5be.el9.x86_64 31/33 2024-01-23T08:33:50.393 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-23T08:33:50.393 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-23T08:33:50.393 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-23T08:33:50.393 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-23T08:33:50.393 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:50.587 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mds-2:17.2.7-397.gb824e5be.el9.x86_64 32/33 2024-01-23T08:33:50.610 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mds-2:17.2.7-397.gb824e5be.el9.x86_64 32/33 2024-01-23T08:33:50.610 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-23T08:33:50.610 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-23T08:33:50.610 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-23T08:33:50.610 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-23T08:33:50.610 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:50.668 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-2:17.2.7-397.gb824e5be.el9.x86_64 33/33 2024-01-23T08:33:52.386 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-2:17.2.7-397.gb824e5be.el9.x86_64 33/33 2024-01-23T08:33:52.386 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-2:17.2.7-397.gb824e5be.el9.x86_64 1/33 2024-01-23T08:33:52.387 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mds-2:17.2.7-397.gb824e5be.el9.x86_64 2/33 2024-01-23T08:33:52.387 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-2:17.2.7-397.gb824e5be.el9.x86_64 3/33 2024-01-23T08:33:52.387 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mon-2:17.2.7-397.gb824e5be.el9.x86_64 4/33 2024-01-23T08:33:52.387 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-osd-2:17.2.7-397.gb824e5be.el9.x86_64 5/33 2024-01-23T08:33:52.387 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-397.gb824e5be.el9.n 6/33 2024-01-23T08:33:52.387 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-01-23T08:33:52.387 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-01-23T08:33:52.387 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-01-23T08:33:52.387 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-01-23T08:33:52.387 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 11/33 2024-01-23T08:33:52.387 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-01-23T08:33:52.387 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-01-23T08:33:52.387 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-01-23T08:33:52.387 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-01-23T08:33:52.388 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-01-23T08:33:52.388 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-01-23T08:33:52.388 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-01-23T08:33:52.388 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-01-23T08:33:52.388 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-01-23T08:33:52.388 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-01-23T08:33:52.388 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-01-23T08:33:52.388 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-01-23T08:33:52.388 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-01-23T08:33:52.388 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-01-23T08:33:52.389 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-01-23T08:33:52.389 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-01-23T08:33:52.389 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-01-23T08:33:52.389 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-01-23T08:33:52.390 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-01-23T08:33:52.390 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-01-23T08:33:52.390 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-01-23T08:33:53.367 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-01-23T08:33:53.367 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:53.368 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-01-23T08:33:53.368 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:33:53.368 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:33:53.368 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:33:53.368 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core-2:17.2.7-397.gb824e5be.el9.noarch 2024-01-23T08:33:53.368 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:33:53.368 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:33:53.368 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-23T08:33:53.368 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-23T08:33:53.368 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-23T08:33:53.368 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-23T08:33:53.368 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-23T08:33:53.368 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-23T08:33:53.368 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-23T08:33:53.369 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-23T08:33:53.369 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-23T08:33:53.369 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-23T08:33:53.369 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-23T08:33:53.369 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-23T08:33:53.369 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-23T08:33:53.369 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-23T08:33:53.369 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-23T08:33:53.369 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ply-3.11-14.el9.noarch 2024-01-23T08:33:53.369 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-23T08:33:53.369 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-23T08:33:53.369 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-23T08:33:53.369 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-23T08:33:53.370 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-23T08:33:53.370 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-23T08:33:53.370 INFO:teuthology.orchestra.run.smithi137.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-23T08:33:53.370 INFO:teuthology.orchestra.run.smithi137.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-23T08:33:53.370 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-23T08:33:53.370 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-23T08:33:53.370 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-23T08:33:53.370 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:53.370 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:33:53.680 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-base 2024-01-23T08:33:54.304 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:54 ago on Tue 23 Jan 2024 08:33:00 AM UTC. 2024-01-23T08:33:54.595 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:33:54.596 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:33:54.596 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:33:54.596 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.596 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:33:54.596 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.596 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:33:54.596 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.596 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:33:54.596 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.596 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:33:54.596 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.596 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:33:54.597 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.597 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:33:54.597 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.597 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:33:54.597 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.597 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:33:54.597 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.597 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:33:54.597 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.597 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:33:54.597 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.597 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:33:54.597 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.597 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:33:54.598 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.598 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:33:54.598 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.598 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:33:54.598 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.598 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:33:54.598 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.598 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:33:54.598 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.598 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:33:54.598 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.598 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:33:54.598 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.598 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:33:54.599 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.599 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:33:54.599 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.599 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:33:54.599 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.599 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:33:54.599 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.599 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:33:54.599 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.599 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:33:54.599 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.599 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:33:54.599 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.599 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:33:54.600 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.600 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:33:54.600 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.600 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:33:54.600 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.600 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:33:54.600 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.600 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:33:54.600 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.600 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:33:54.600 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.600 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:33:54.600 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.600 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:33:54.601 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.601 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:33:54.601 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.601 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:33:54.601 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.601 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:33:54.601 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.601 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.601 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.601 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:33:54.601 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.601 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.601 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.601 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.602 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:33:54.602 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.602 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.602 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.602 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.602 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.602 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.602 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:33:54.602 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.602 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.602 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.602 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.602 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:33:54.602 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.602 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.603 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.603 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.603 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:33:54.603 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.603 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.603 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.603 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.603 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:33:54.603 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.603 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.603 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.603 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.603 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:33:54.604 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.604 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.604 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.604 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.604 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:33:54.604 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.604 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.604 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.604 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:33:54.605 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-base-2:17.2.7-397.gb824e5be.el9.x86_64 is already installed. 2024-01-23T08:33:54.663 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:33:54.664 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-01-23T08:33:54.664 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:33:54.732 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install cephadm 2024-01-23T08:33:55.325 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:55 ago on Tue 23 Jan 2024 08:33:00 AM UTC. 2024-01-23T08:33:55.618 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:33:55.618 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:33:55.618 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:33:55.618 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.618 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:33:55.618 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.618 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:33:55.619 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.619 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:33:55.619 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.619 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:33:55.619 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.619 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:33:55.619 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.619 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:33:55.619 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.619 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:33:55.619 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.619 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:33:55.619 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.620 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:33:55.620 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.620 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:33:55.620 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.620 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:33:55.620 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.620 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:33:55.620 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.620 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:33:55.620 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.620 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:33:55.620 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.621 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:33:55.621 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.621 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:33:55.621 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.621 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:33:55.621 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.621 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:33:55.621 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.621 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:33:55.621 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.621 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:33:55.621 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.621 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:33:55.621 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.622 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:33:55.622 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.622 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:33:55.622 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.622 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:33:55.622 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.622 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:33:55.622 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.622 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:33:55.622 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.622 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:33:55.622 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.622 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:33:55.622 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.623 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:33:55.623 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.623 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:33:55.623 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.623 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:33:55.623 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.623 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:33:55.623 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.623 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:33:55.623 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.623 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:33:55.623 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.623 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:33:55.623 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.623 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:33:55.624 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.624 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.624 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.624 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:33:55.624 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.624 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.624 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.624 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.624 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:33:55.624 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.624 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.624 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.624 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.624 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.625 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.625 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:33:55.625 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.625 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.625 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.625 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.625 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:33:55.625 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.625 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.625 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.625 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.625 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:33:55.625 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.625 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.625 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.626 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.626 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:33:55.626 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.626 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.626 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.626 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.626 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:33:55.626 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.626 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.626 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.626 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.626 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:33:55.626 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.626 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.626 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.627 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:33:55.684 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:33:55.685 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:33:55.685 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-01-23T08:33:55.685 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:33:55.685 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-01-23T08:33:55.685 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm noarch 2:17.2.7-397.gb824e5be.el9 ceph-noarch 80 k 2024-01-23T08:33:55.685 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:55.685 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-01-23T08:33:55.685 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:33:55.685 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-01-23T08:33:55.685 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:55.686 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 80 k 2024-01-23T08:33:55.686 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 364 k 2024-01-23T08:33:55.686 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-01-23T08:33:56.029 INFO:teuthology.orchestra.run.smithi137.stdout:cephadm-17.2.7-397.gb824e5be.el9.noarch.rpm 235 kB/s | 80 kB 00:00 2024-01-23T08:33:56.029 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:33:56.029 INFO:teuthology.orchestra.run.smithi137.stdout:Total 234 kB/s | 80 kB 00:00 2024-01-23T08:33:56.030 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-01-23T08:33:56.034 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-01-23T08:33:56.034 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-01-23T08:33:56.036 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-01-23T08:33:56.036 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-01-23T08:33:56.115 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-01-23T08:33:56.567 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:17.2.7-397.gb824e5be.el9.noarch 1/1 2024-01-23T08:33:56.675 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : cephadm-2:17.2.7-397.gb824e5be.el9.noarch 1/1 2024-01-23T08:33:57.112 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:17.2.7-397.gb824e5be.el9.noarch 1/1 2024-01-23T08:33:57.454 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : cephadm-2:17.2.7-397.gb824e5be.el9.noarch 1/1 2024-01-23T08:33:57.455 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:57.455 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-01-23T08:33:57.455 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm-2:17.2.7-397.gb824e5be.el9.noarch 2024-01-23T08:33:57.455 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:57.455 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:33:57.667 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-immutable-object-cache 2024-01-23T08:33:58.272 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:58 ago on Tue 23 Jan 2024 08:33:00 AM UTC. 2024-01-23T08:33:58.564 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:33:58.564 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:33:58.564 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:33:58.564 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.564 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:33:58.565 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.565 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:33:58.565 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.565 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:33:58.565 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.565 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:33:58.565 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.565 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:33:58.565 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.565 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:33:58.565 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.565 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:33:58.565 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.565 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:33:58.566 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.566 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:33:58.566 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.566 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:33:58.566 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.566 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:33:58.566 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.566 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:33:58.566 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.566 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:33:58.566 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.566 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:33:58.566 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.566 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:33:58.567 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.567 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:33:58.567 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.567 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:33:58.567 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.567 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:33:58.567 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.567 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:33:58.567 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.567 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:33:58.567 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.567 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:33:58.567 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.568 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:33:58.568 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.568 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:33:58.568 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.568 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:33:58.568 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.568 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:33:58.568 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.568 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:33:58.568 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.568 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:33:58.568 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.569 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:33:58.569 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.569 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:33:58.569 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.569 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:33:58.569 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.569 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:33:58.569 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.569 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:33:58.569 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.569 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:33:58.569 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.569 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:33:58.570 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.570 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:33:58.570 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.570 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:33:58.570 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.570 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.570 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.570 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:33:58.570 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.570 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.570 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.570 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.570 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:33:58.570 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.571 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.571 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.571 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.571 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.571 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.571 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:33:58.571 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.571 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.571 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.571 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.571 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:33:58.571 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.571 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.571 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.571 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.572 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:33:58.572 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.572 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.572 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.572 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.572 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:33:58.572 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.572 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.572 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.572 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.572 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:33:58.572 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.572 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.572 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.572 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.572 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:33:58.573 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.573 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.573 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.573 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:33:58.629 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:33:58.630 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:33:58.630 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-01-23T08:33:58.630 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:33:58.630 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-01-23T08:33:58.630 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-397.gb824e5be.el9 ceph 137 k 2024-01-23T08:33:58.630 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:58.630 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-01-23T08:33:58.630 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:33:58.631 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-01-23T08:33:58.631 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:33:58.631 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 137 k 2024-01-23T08:33:58.631 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 379 k 2024-01-23T08:33:58.631 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-01-23T08:33:58.859 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-immutable-object-cache-17.2.7-397.gb824e5b 601 kB/s | 137 kB 00:00 2024-01-23T08:33:58.860 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:33:58.860 INFO:teuthology.orchestra.run.smithi137.stdout:Total 598 kB/s | 137 kB 00:00 2024-01-23T08:33:58.860 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-01-23T08:33:58.867 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-01-23T08:33:58.867 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-01-23T08:33:58.919 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-01-23T08:33:58.920 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-01-23T08:33:59.054 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-01-23T08:33:59.127 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-397.gb824e5be.e 1/1 2024-01-23T08:33:59.152 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-397.gb824e5be.e 1/1 2024-01-23T08:33:59.152 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-23T08:33:59.152 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-23T08:33:59.152 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:00.235 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-397.gb824e5be.e 1/1 2024-01-23T08:34:00.235 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:00.236 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-01-23T08:34:00.236 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:34:00.236 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:00.236 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:34:00.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr 2024-01-23T08:34:01.105 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:01 ago on Tue 23 Jan 2024 08:33:00 AM UTC. 2024-01-23T08:34:01.396 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:34:01.397 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:34:01.397 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:34:01.397 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.397 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:34:01.397 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.397 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:34:01.397 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.397 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:34:01.397 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.397 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:34:01.397 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.397 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:34:01.397 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.397 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:34:01.398 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.398 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:34:01.398 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.398 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:34:01.398 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.398 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:34:01.398 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.398 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:34:01.398 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.398 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:34:01.398 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.398 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:34:01.398 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.398 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:34:01.399 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.399 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:34:01.399 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.399 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:34:01.399 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.399 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:34:01.399 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.399 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:34:01.399 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.399 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:34:01.399 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.399 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:34:01.399 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.399 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:34:01.399 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.400 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:34:01.400 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.400 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:34:01.400 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.400 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:34:01.400 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.400 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:34:01.400 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.400 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:34:01.400 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.400 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:34:01.400 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.400 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:34:01.400 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.401 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:34:01.401 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.401 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:34:01.401 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.401 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:34:01.401 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.401 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:34:01.401 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.401 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:34:01.401 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.401 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:34:01.401 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.401 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:34:01.401 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.402 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:34:01.402 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.402 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:34:01.402 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.402 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.402 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.402 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:34:01.402 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.402 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.402 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.402 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.402 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:34:01.403 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.403 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.403 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.403 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.403 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.403 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.403 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:34:01.403 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.403 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.403 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.403 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.403 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:34:01.404 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.404 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.404 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.404 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.404 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:34:01.404 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.404 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.404 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.404 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.404 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:34:01.404 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.404 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.404 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.405 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.405 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:34:01.405 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.405 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.405 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.405 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.405 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:34:01.405 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.405 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.405 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.405 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:34:01.406 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-mgr-2:17.2.7-397.gb824e5be.el9.x86_64 is already installed. 2024-01-23T08:34:01.464 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:34:01.465 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-01-23T08:34:01.465 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:34:01.526 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-dashboard 2024-01-23T08:34:02.122 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:02 ago on Tue 23 Jan 2024 08:33:00 AM UTC. 2024-01-23T08:34:02.432 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:34:02.432 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:34:02.432 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:34:02.432 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.432 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:34:02.432 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.432 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:34:02.432 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.433 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:34:02.433 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.433 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:34:02.433 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.433 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:34:02.433 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.433 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:34:02.433 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.433 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:34:02.433 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.433 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:34:02.433 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.433 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:34:02.433 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.434 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:34:02.434 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.434 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:34:02.434 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.434 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:34:02.434 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.434 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:34:02.434 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.434 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:34:02.434 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.434 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:34:02.434 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.434 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:34:02.434 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.435 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:34:02.435 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.435 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:34:02.435 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.435 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:34:02.435 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.435 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:34:02.435 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.435 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:34:02.435 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.435 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:34:02.435 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.435 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:34:02.436 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.436 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:34:02.436 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.436 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:34:02.436 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.436 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:34:02.436 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.436 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:34:02.436 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.436 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:34:02.436 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.436 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:34:02.436 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.437 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:34:02.437 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.437 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:34:02.437 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.437 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:34:02.437 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.437 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:34:02.437 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.437 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:34:02.437 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.437 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:34:02.437 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.437 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:34:02.437 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.438 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.438 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.438 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:34:02.438 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.438 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.438 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.438 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.438 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:34:02.438 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.438 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.438 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.438 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.438 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.438 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.439 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:34:02.439 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.439 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.439 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.439 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.439 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:34:02.439 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.439 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.439 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.439 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.439 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:34:02.439 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.439 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.439 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.439 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.440 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:34:02.440 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.440 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.440 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.440 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.440 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:34:02.440 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.440 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.440 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.440 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.440 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:34:02.440 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.440 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.441 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.441 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:34:02.501 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:34:02.502 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:02.502 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-01-23T08:34:02.502 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:02.502 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-01-23T08:34:02.502 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard noarch 2:17.2.7-397.gb824e5be.el9 ceph-noarch 1.7 M 2024-01-23T08:34:02.503 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-01-23T08:34:02.503 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards noarch 2:17.2.7-397.gb824e5be.el9 ceph-noarch 23 k 2024-01-23T08:34:02.503 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts noarch 2:17.2.7-397.gb824e5be.el9 ceph-noarch 15 k 2024-01-23T08:34:02.503 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-01-23T08:34:02.503 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-23T08:34:02.503 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-23T08:34:02.503 INFO:teuthology.orchestra.run.smithi137.stdout:Installing weak dependencies: 2024-01-23T08:34:02.503 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-01-23T08:34:02.503 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:02.503 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-01-23T08:34:02.503 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:02.503 INFO:teuthology.orchestra.run.smithi137.stdout:Install 7 Packages 2024-01-23T08:34:02.503 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:02.504 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 2.0 M 2024-01-23T08:34:02.504 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 8.1 M 2024-01-23T08:34:02.504 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-01-23T08:34:03.048 INFO:teuthology.orchestra.run.smithi137.stdout:(1/7): ceph-prometheus-alerts-17.2.7-397.gb824e 76 kB/s | 15 kB 00:00 2024-01-23T08:34:03.164 INFO:teuthology.orchestra.run.smithi137.stdout:(2/7): ceph-grafana-dashboards-17.2.7-397.gb824 76 kB/s | 23 kB 00:00 2024-01-23T08:34:03.223 INFO:teuthology.orchestra.run.smithi137.stdout:(3/7): ceph-mgr-dashboard-17.2.7-397.gb824e5be. 4.6 MB/s | 1.7 MB 00:00 2024-01-23T08:34:03.274 INFO:teuthology.orchestra.run.smithi137.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 40 kB/s | 9.0 kB 00:00 2024-01-23T08:34:03.474 INFO:teuthology.orchestra.run.smithi137.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 132 kB/s | 41 kB 00:00 2024-01-23T08:34:03.499 INFO:teuthology.orchestra.run.smithi137.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 122 kB/s | 31 kB 00:00 2024-01-23T08:34:03.540 INFO:teuthology.orchestra.run.smithi137.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 706 kB/s | 188 kB 00:00 2024-01-23T08:34:03.544 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:34:03.545 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-01-23T08:34:03.545 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-01-23T08:34:03.553 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-01-23T08:34:03.553 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-01-23T08:34:03.620 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-01-23T08:34:03.621 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-01-23T08:34:03.951 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-01-23T08:34:04.008 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-01-23T08:34:04.043 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-01-23T08:34:04.089 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-01-23T08:34:04.129 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-01-23T08:34:04.158 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-397.gb824e5be.el9.no 5/7 2024-01-23T08:34:04.388 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-397.gb824e5be.el9.n 6/7 2024-01-23T08:34:04.419 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-397.gb824e5be.el9.noarch 7/7 2024-01-23T08:34:04.987 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-397.gb824e5be.el9.noarch 7/7 2024-01-23T08:34:04.987 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-397.gb824e5be.el9.n 1/7 2024-01-23T08:34:04.987 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-397.gb824e5be.el9.noarch 2/7 2024-01-23T08:34:04.987 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-397.gb824e5be.el9.no 3/7 2024-01-23T08:34:04.987 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-01-23T08:34:04.987 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-01-23T08:34:04.987 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-01-23T08:34:05.331 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-01-23T08:34:05.331 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:05.331 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-01-23T08:34:05.331 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards-2:17.2.7-397.gb824e5be.el9.noarch 2024-01-23T08:34:05.331 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard-2:17.2.7-397.gb824e5be.el9.noarch 2024-01-23T08:34:05.331 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts-2:17.2.7-397.gb824e5be.el9.noarch 2024-01-23T08:34:05.331 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-01-23T08:34:05.331 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-01-23T08:34:05.331 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-23T08:34:05.331 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-23T08:34:05.331 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:05.331 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:34:05.570 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-23T08:34:06.171 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:06 ago on Tue 23 Jan 2024 08:33:00 AM UTC. 2024-01-23T08:34:06.463 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:34:06.463 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:34:06.464 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:34:06.464 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.464 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:34:06.464 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.464 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:34:06.464 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.464 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:34:06.464 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.464 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:34:06.464 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.464 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:34:06.464 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.464 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:34:06.465 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.465 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:34:06.465 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.465 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:34:06.465 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.465 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:34:06.465 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.465 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:34:06.465 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.465 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:34:06.465 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.465 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:34:06.465 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.465 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:34:06.465 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.466 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:34:06.466 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.466 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:34:06.466 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.466 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:34:06.466 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.466 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:34:06.466 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.466 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:34:06.466 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.466 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:34:06.466 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.466 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:34:06.466 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.466 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:34:06.467 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.467 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:34:06.467 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.467 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:34:06.467 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.467 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:34:06.467 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.467 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:34:06.467 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.467 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:34:06.467 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.467 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:34:06.467 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.467 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:34:06.468 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.468 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:34:06.468 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.468 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:34:06.468 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.468 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:34:06.468 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.468 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:34:06.468 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.468 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:34:06.468 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.468 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:34:06.468 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.468 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:34:06.469 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.469 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:34:06.469 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.469 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.469 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.469 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:34:06.469 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.469 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.469 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.469 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.469 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:34:06.469 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.469 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.469 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.469 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.470 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.470 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.470 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:34:06.470 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.470 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.470 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.470 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.470 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:34:06.470 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.470 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.470 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.470 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.470 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:34:06.470 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.471 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.471 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.471 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.471 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:34:06.471 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.471 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.471 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.471 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.471 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:34:06.471 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.471 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.471 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.471 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.471 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:34:06.471 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.472 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.472 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.472 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:34:06.532 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:34:06.533 INFO:teuthology.orchestra.run.smithi137.stdout:====================================================================================== 2024-01-23T08:34:06.533 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-01-23T08:34:06.533 INFO:teuthology.orchestra.run.smithi137.stdout:====================================================================================== 2024-01-23T08:34:06.533 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-01-23T08:34:06.533 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-397.gb824e5be.el9 ceph-noarch 7.4 M 2024-01-23T08:34:06.533 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-01-23T08:34:06.534 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-01-23T08:34:06.534 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-01-23T08:34:06.534 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-01-23T08:34:06.534 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-01-23T08:34:06.534 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-01-23T08:34:06.534 INFO:teuthology.orchestra.run.smithi137.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-01-23T08:34:06.534 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-01-23T08:34:06.534 INFO:teuthology.orchestra.run.smithi137.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-23T08:34:06.534 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-01-23T08:34:06.534 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-01-23T08:34:06.534 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-01-23T08:34:06.534 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:06.534 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-01-23T08:34:06.534 INFO:teuthology.orchestra.run.smithi137.stdout:====================================================================================== 2024-01-23T08:34:06.535 INFO:teuthology.orchestra.run.smithi137.stdout:Install 12 Packages 2024-01-23T08:34:06.535 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:06.535 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 38 M 2024-01-23T08:34:06.535 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 209 M 2024-01-23T08:34:06.535 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-01-23T08:34:06.881 INFO:teuthology.orchestra.run.smithi137.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-01-23T08:34:06.923 INFO:teuthology.orchestra.run.smithi137.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 802 kB/s | 33 kB 00:00 2024-01-23T08:34:06.948 INFO:teuthology.orchestra.run.smithi137.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.7 MB/s | 803 kB 00:00 2024-01-23T08:34:06.989 INFO:teuthology.orchestra.run.smithi137.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 437 kB/s | 18 kB 00:00 2024-01-23T08:34:07.032 INFO:teuthology.orchestra.run.smithi137.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 840 kB/s | 35 kB 00:00 2024-01-23T08:34:07.082 INFO:teuthology.orchestra.run.smithi137.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 19 MB/s | 3.0 MB 00:00 2024-01-23T08:34:07.132 INFO:teuthology.orchestra.run.smithi137.stdout:(7/12): python3-devel-3.9.17-2.el9.x86_64.rpm 4.8 MB/s | 245 kB 00:00 2024-01-23T08:34:07.199 INFO:teuthology.orchestra.run.smithi137.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2024-01-23T08:34:07.249 INFO:teuthology.orchestra.run.smithi137.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.9 MB/s | 250 kB 00:00 2024-01-23T08:34:07.316 INFO:teuthology.orchestra.run.smithi137.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 28 MB/s | 5.1 MB 00:00 2024-01-23T08:34:07.567 INFO:teuthology.orchestra.run.smithi137.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 51 MB/s | 16 MB 00:00 2024-01-23T08:34:07.717 INFO:teuthology.orchestra.run.smithi137.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-3 7.5 MB/s | 7.4 MB 00:00 2024-01-23T08:34:07.717 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:34:07.718 INFO:teuthology.orchestra.run.smithi137.stdout:Total 32 MB/s | 38 MB 00:01 2024-01-23T08:34:08.072 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-01-23T08:34:08.096 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-01-23T08:34:08.096 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-01-23T08:34:08.614 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-01-23T08:34:08.614 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-01-23T08:34:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-01-23T08:34:09.439 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-01-23T08:34:09.492 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-01-23T08:34:09.588 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-01-23T08:34:09.624 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-01-23T08:34:09.801 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-23T08:34:09.831 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-23T08:34:09.911 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-01-23T08:34:10.566 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-01-23T08:34:10.677 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-01-23T08:34:11.922 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-01-23T08:34:12.988 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-01-23T08:34:13.014 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-397.gb824e5 12/12 2024-01-23T08:34:15.561 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-397.gb824e5 12/12 2024-01-23T08:34:15.561 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-397.gb824e5 1/12 2024-01-23T08:34:15.561 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-01-23T08:34:15.561 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-01-23T08:34:15.561 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-01-23T08:34:15.561 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-01-23T08:34:15.561 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-01-23T08:34:15.561 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-01-23T08:34:15.561 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-01-23T08:34:15.561 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 9/12 2024-01-23T08:34:15.561 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-01-23T08:34:15.562 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-01-23T08:34:16.027 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-01-23T08:34:16.027 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:16.027 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-01-23T08:34:16.028 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local-2:17.2.7-397.gb824e5be.el9.noarch 2024-01-23T08:34:16.028 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-23T08:34:16.028 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-23T08:34:16.028 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-23T08:34:16.028 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-23T08:34:16.028 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-23T08:34:16.028 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-23T08:34:16.028 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-23T08:34:16.028 INFO:teuthology.orchestra.run.smithi137.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-23T08:34:16.028 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-23T08:34:16.028 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-23T08:34:16.028 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-23T08:34:16.028 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:16.029 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:34:16.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-rook 2024-01-23T08:34:17.023 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:17 ago on Tue 23 Jan 2024 08:33:00 AM UTC. 2024-01-23T08:34:17.314 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:34:17.315 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:34:17.315 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:34:17.315 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.315 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:34:17.315 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.315 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:34:17.315 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.315 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:34:17.315 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.315 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:34:17.315 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.316 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:34:17.316 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.316 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:34:17.316 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.316 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:34:17.316 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.316 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:34:17.316 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.316 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:34:17.316 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.316 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:34:17.316 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.316 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:34:17.317 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.317 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:34:17.317 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.317 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:34:17.317 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.317 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:34:17.317 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.317 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:34:17.317 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.317 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:34:17.317 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.317 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:34:17.317 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.317 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:34:17.317 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.318 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:34:17.318 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.318 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:34:17.318 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.318 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:34:17.318 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.318 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:34:17.318 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.318 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:34:17.318 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.318 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:34:17.318 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.318 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:34:17.318 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.319 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:34:17.319 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.319 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:34:17.319 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.319 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:34:17.319 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.319 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:34:17.319 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.319 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:34:17.319 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.319 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:34:17.319 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.319 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:34:17.319 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.319 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:34:17.320 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.320 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:34:17.320 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.320 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:34:17.320 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.320 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:34:17.320 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.320 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.320 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.320 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:34:17.320 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.320 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.320 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.321 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.321 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:34:17.321 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.321 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.321 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.321 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.321 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.321 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.321 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:34:17.321 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.321 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.321 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.321 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.321 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:34:17.322 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.322 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.322 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.322 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.322 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:34:17.322 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.322 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.322 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.322 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.322 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:34:17.322 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.322 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.322 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.322 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.322 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:34:17.323 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.323 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.323 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.323 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.323 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:34:17.323 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.323 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.323 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.323 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:34:17.382 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:34:17.383 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:17.383 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-01-23T08:34:17.384 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:17.384 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-01-23T08:34:17.384 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook noarch 2:17.2.7-397.gb824e5be.el9 ceph-noarch 47 k 2024-01-23T08:34:17.384 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-01-23T08:34:17.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-23T08:34:17.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-23T08:34:17.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth noarch 1:2.26.2-1.el9 epel 202 k 2024-01-23T08:34:17.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-01-23T08:34:17.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-01-23T08:34:17.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-23T08:34:17.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-01-23T08:34:17.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-01-23T08:34:17.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-01-23T08:34:17.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-01-23T08:34:17.384 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-23T08:34:17.385 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-23T08:34:17.385 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:17.385 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-01-23T08:34:17.385 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:17.385 INFO:teuthology.orchestra.run.smithi137.stdout:Install 13 Packages 2024-01-23T08:34:17.385 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:17.385 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 2.2 M 2024-01-23T08:34:17.386 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 26 M 2024-01-23T08:34:17.386 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-01-23T08:34:17.873 INFO:teuthology.orchestra.run.smithi137.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 95 kB/s | 26 kB 00:00 2024-01-23T08:34:17.899 INFO:teuthology.orchestra.run.smithi137.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 65 kB/s | 19 kB 00:00 2024-01-23T08:34:17.924 INFO:teuthology.orchestra.run.smithi137.stdout:(3/13): ceph-mgr-rook-17.2.7-397.gb824e5be.el9. 148 kB/s | 47 kB 00:00 2024-01-23T08:34:18.174 INFO:teuthology.orchestra.run.smithi137.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 578 kB/s | 159 kB 00:00 2024-01-23T08:34:18.266 INFO:teuthology.orchestra.run.smithi137.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 605 kB/s | 222 kB 00:00 2024-01-23T08:34:18.291 INFO:teuthology.orchestra.run.smithi137.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 459 kB/s | 54 kB 00:00 2024-01-23T08:34:18.508 INFO:teuthology.orchestra.run.smithi137.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 477 kB/s | 279 kB 00:00 2024-01-23T08:34:18.533 INFO:teuthology.orchestra.run.smithi137.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 120 kB/s | 32 kB 00:00 2024-01-23T08:34:18.559 INFO:teuthology.orchestra.run.smithi137.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 56 kB/s | 15 kB 00:00 2024-01-23T08:34:18.651 INFO:teuthology.orchestra.run.smithi137.stdout:(10/13): python3-google-auth-2.26.2-1.el9.noarc 1.4 MB/s | 202 kB 00:00 2024-01-23T08:34:18.759 INFO:teuthology.orchestra.run.smithi137.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 4.5 MB/s | 1.0 MB 00:00 2024-01-23T08:34:18.793 INFO:teuthology.orchestra.run.smithi137.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 253 kB/s | 59 kB 00:00 2024-01-23T08:34:18.818 INFO:teuthology.orchestra.run.smithi137.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 538 kB/s | 90 kB 00:00 2024-01-23T08:34:18.820 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:34:18.820 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.5 MB/s | 2.2 MB 00:01 2024-01-23T08:34:18.882 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-01-23T08:34:18.897 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-01-23T08:34:18.897 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-01-23T08:34:18.998 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-01-23T08:34:18.999 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-01-23T08:34:19.526 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-01-23T08:34:19.697 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-01-23T08:34:19.752 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-01-23T08:34:19.807 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-01-23T08:34:19.845 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-01-23T08:34:19.886 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-01-23T08:34:19.979 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-01-23T08:34:20.089 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-google-auth-1:2.26.2-1.el9.noarch 7/13 2024-01-23T08:34:20.138 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-01-23T08:34:20.855 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-01-23T08:34:20.918 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-01-23T08:34:20.943 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-01-23T08:34:20.989 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-01-23T08:34:21.022 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-rook-2:17.2.7-397.gb824e5be.el9.noarch 13/13 2024-01-23T08:34:21.819 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-397.gb824e5be.el9.noarch 13/13 2024-01-23T08:34:21.819 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-rook-2:17.2.7-397.gb824e5be.el9.noarch 1/13 2024-01-23T08:34:21.819 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-01-23T08:34:21.819 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-01-23T08:34:21.819 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-01-23T08:34:21.819 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-01-23T08:34:21.819 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-01-23T08:34:21.819 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-01-23T08:34:21.820 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-01-23T08:34:21.820 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-01-23T08:34:21.820 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-google-auth-1:2.26.2-1.el9.noarch 10/13 2024-01-23T08:34:21.820 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-01-23T08:34:21.820 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-01-23T08:34:22.209 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-01-23T08:34:22.209 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:22.209 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-01-23T08:34:22.209 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook-2:17.2.7-397.gb824e5be.el9.noarch 2024-01-23T08:34:22.209 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-23T08:34:22.209 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-23T08:34:22.209 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth-1:2.26.2-1.el9.noarch 2024-01-23T08:34:22.209 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-23T08:34:22.209 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-23T08:34:22.209 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-23T08:34:22.209 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-23T08:34:22.210 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-23T08:34:22.210 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-23T08:34:22.210 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-23T08:34:22.210 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-23T08:34:22.210 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-23T08:34:22.210 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:22.210 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:34:22.448 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-cephadm 2024-01-23T08:34:23.049 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:23 ago on Tue 23 Jan 2024 08:33:00 AM UTC. 2024-01-23T08:34:23.340 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:34:23.340 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:34:23.340 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:34:23.340 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.340 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:34:23.340 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.340 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:34:23.340 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.341 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:34:23.341 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.341 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:34:23.341 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.341 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:34:23.341 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.341 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:34:23.341 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.341 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:34:23.341 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.341 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:34:23.341 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.341 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:34:23.341 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.341 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:34:23.341 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.342 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:34:23.342 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.342 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:34:23.342 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.342 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:34:23.342 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.342 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:34:23.342 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.342 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:34:23.342 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.342 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:34:23.342 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.342 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:34:23.342 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.342 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:34:23.342 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.343 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:34:23.343 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.343 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:34:23.343 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.343 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:34:23.343 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.343 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:34:23.343 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.343 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:34:23.343 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.343 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:34:23.343 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.343 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:34:23.343 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.343 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:34:23.344 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.344 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:34:23.344 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.344 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:34:23.344 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.344 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:34:23.344 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.344 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:34:23.344 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.344 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:34:23.344 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.344 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:34:23.344 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.344 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:34:23.344 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.345 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:34:23.345 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.345 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:34:23.345 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.345 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:34:23.345 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.345 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.345 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.345 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:34:23.345 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.345 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.345 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.345 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.345 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:34:23.345 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.346 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.346 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.346 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.346 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.346 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.346 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:34:23.346 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.346 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.346 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.346 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.346 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:34:23.346 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.346 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.346 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.346 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.347 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:34:23.347 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.347 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.347 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.347 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.347 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:34:23.347 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.347 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.347 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.347 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.347 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:34:23.347 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.347 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.347 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.348 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.348 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:34:23.348 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.348 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.348 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.348 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:34:23.406 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:34:23.407 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:23.407 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-01-23T08:34:23.408 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:23.408 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-01-23T08:34:23.408 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm noarch 2:17.2.7-397.gb824e5be.el9 ceph-noarch 121 k 2024-01-23T08:34:23.408 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-01-23T08:34:23.408 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-23T08:34:23.408 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-01-23T08:34:23.408 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2 noarch 2.11.3-4.el9 appstream 249 k 2024-01-23T08:34:23.408 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-23T08:34:23.408 INFO:teuthology.orchestra.run.smithi137.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-23T08:34:23.408 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:23.408 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-01-23T08:34:23.408 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:23.408 INFO:teuthology.orchestra.run.smithi137.stdout:Install 6 Packages 2024-01-23T08:34:23.408 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:23.409 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 7.0 M 2024-01-23T08:34:23.409 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 33 M 2024-01-23T08:34:23.409 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-01-23T08:34:24.044 INFO:teuthology.orchestra.run.smithi137.stdout:(1/6): ceph-mgr-cephadm-17.2.7-397.gb824e5be.el 571 kB/s | 121 kB 00:00 2024-01-23T08:34:24.336 INFO:teuthology.orchestra.run.smithi137.stdout:(2/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 495 kB/s | 249 kB 00:00 2024-01-23T08:34:24.587 INFO:teuthology.orchestra.run.smithi137.stdout:(3/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-01-23T08:34:24.653 INFO:teuthology.orchestra.run.smithi137.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 182 kB/s | 58 kB 00:00 2024-01-23T08:34:24.687 INFO:teuthology.orchestra.run.smithi137.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 510 kB/s | 51 kB 00:00 2024-01-23T08:34:25.596 INFO:teuthology.orchestra.run.smithi137.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.4 MB/s | 6.0 MB 00:01 2024-01-23T08:34:25.596 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:34:25.596 INFO:teuthology.orchestra.run.smithi137.stdout:Total 3.2 MB/s | 7.0 MB 00:02 2024-01-23T08:34:25.670 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-01-23T08:34:25.682 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-01-23T08:34:25.682 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-01-23T08:34:25.787 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-01-23T08:34:25.787 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-01-23T08:34:26.090 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-01-23T08:34:26.258 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-01-23T08:34:26.305 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-01-23T08:34:26.812 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-01-23T08:34:26.898 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-01-23T08:34:26.962 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-01-23T08:34:26.992 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-397.gb824e5be.el9.noarch 6/6 2024-01-23T08:34:27.791 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-397.gb824e5be.el9.noarch 6/6 2024-01-23T08:34:27.791 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-397.gb824e5be.el9.noarch 1/6 2024-01-23T08:34:27.791 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-01-23T08:34:27.792 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-01-23T08:34:27.792 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-01-23T08:34:27.792 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-01-23T08:34:28.121 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-01-23T08:34:28.121 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:28.121 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-01-23T08:34:28.122 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm-2:17.2.7-397.gb824e5be.el9.noarch 2024-01-23T08:34:28.122 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-23T08:34:28.122 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-23T08:34:28.122 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-23T08:34:28.122 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-23T08:34:28.122 INFO:teuthology.orchestra.run.smithi137.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-23T08:34:28.122 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:28.122 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:34:28.432 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-fuse 2024-01-23T08:34:29.030 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:29 ago on Tue 23 Jan 2024 08:33:00 AM UTC. 2024-01-23T08:34:29.319 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:34:29.320 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:34:29.320 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:34:29.320 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.320 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:34:29.320 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.320 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:34:29.320 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.320 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:34:29.320 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.320 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:34:29.321 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.321 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:34:29.321 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.321 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:34:29.321 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.321 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:34:29.321 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.321 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:34:29.321 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.321 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:34:29.321 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.321 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:34:29.321 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.321 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:34:29.322 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.322 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:34:29.322 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.322 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:34:29.322 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.322 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:34:29.322 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.322 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:34:29.322 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.322 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:34:29.322 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.322 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:34:29.322 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.322 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:34:29.323 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.323 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:34:29.323 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.323 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:34:29.323 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.323 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:34:29.323 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.323 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:34:29.323 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.323 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:34:29.323 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.323 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:34:29.323 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.323 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:34:29.324 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.324 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:34:29.324 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.324 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:34:29.324 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.324 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:34:29.324 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.324 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:34:29.324 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.324 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:34:29.324 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.324 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:34:29.324 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.325 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:34:29.325 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.325 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:34:29.325 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.325 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:34:29.325 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.325 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:34:29.325 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.325 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:34:29.325 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.325 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.325 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.325 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:34:29.325 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.325 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.326 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.326 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.326 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:34:29.326 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.326 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.326 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.326 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.326 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.326 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.326 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:34:29.326 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.326 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.326 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.326 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.327 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:34:29.327 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.327 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.327 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.327 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.327 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:34:29.327 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.327 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.327 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.327 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.327 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:34:29.327 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.328 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.328 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.328 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.328 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:34:29.328 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.328 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.328 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.328 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.328 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:34:29.328 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.328 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.328 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.328 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:34:29.387 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:34:29.388 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-01-23T08:34:29.388 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:29.388 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-01-23T08:34:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse x86_64 2:17.2.7-397.gb824e5be.el9 ceph 810 k 2024-01-23T08:34:29.388 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-01-23T08:34:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-01-23T08:34:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:29.388 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-01-23T08:34:29.388 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:29.389 INFO:teuthology.orchestra.run.smithi137.stdout:Install 2 Packages 2024-01-23T08:34:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:29.389 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 891 k 2024-01-23T08:34:29.389 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 2.6 M 2024-01-23T08:34:29.389 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-01-23T08:34:29.805 INFO:teuthology.orchestra.run.smithi137.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 229 kB/s | 80 kB 00:00 2024-01-23T08:34:29.881 INFO:teuthology.orchestra.run.smithi137.stdout:(2/2): ceph-fuse-17.2.7-397.gb824e5be.el9.x86_6 1.9 MB/s | 810 kB 00:00 2024-01-23T08:34:29.881 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:34:29.881 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.8 MB/s | 891 kB 00:00 2024-01-23T08:34:29.893 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-01-23T08:34:29.914 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-01-23T08:34:29.914 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-01-23T08:34:29.994 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-01-23T08:34:29.994 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-01-23T08:34:30.180 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-01-23T08:34:30.309 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-01-23T08:34:30.372 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-fuse-2:17.2.7-397.gb824e5be.el9.x86_64 2/2 2024-01-23T08:34:31.058 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-fuse-2:17.2.7-397.gb824e5be.el9.x86_64 2/2 2024-01-23T08:34:31.058 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-fuse-2:17.2.7-397.gb824e5be.el9.x86_64 1/2 2024-01-23T08:34:31.393 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-01-23T08:34:31.393 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:31.393 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-01-23T08:34:31.394 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse-2:17.2.7-397.gb824e5be.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-01-23T08:34:31.394 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:31.394 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:34:31.655 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados-devel 2024-01-23T08:34:32.271 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:32 ago on Tue 23 Jan 2024 08:33:00 AM UTC. 2024-01-23T08:34:32.562 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:34:32.562 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:34:32.562 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:34:32.562 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.562 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:34:32.562 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.563 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:34:32.563 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.563 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:34:32.563 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.563 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:34:32.563 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.563 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:34:32.563 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.563 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:34:32.563 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.563 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:34:32.563 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.563 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:34:32.563 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.564 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:34:32.564 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.564 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:34:32.564 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.564 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:34:32.564 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.564 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:34:32.564 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.564 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:34:32.564 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.564 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:34:32.564 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.564 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:34:32.565 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.565 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:34:32.565 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.565 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:34:32.565 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.565 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:34:32.565 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.565 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:34:32.565 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.565 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:34:32.565 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.565 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:34:32.565 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.565 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:34:32.565 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.566 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:34:32.566 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.566 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:34:32.566 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.566 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:34:32.566 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.566 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:34:32.566 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.566 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:34:32.566 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.566 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:34:32.566 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.566 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:34:32.566 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.567 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:34:32.567 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.567 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:34:32.567 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.567 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:34:32.567 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.567 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:34:32.567 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.567 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:34:32.567 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.567 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:34:32.567 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.567 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:34:32.567 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.567 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.568 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.568 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:34:32.568 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.568 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.568 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.568 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.568 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:34:32.568 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.568 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.568 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.568 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.568 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.568 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.569 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:34:32.569 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.569 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.569 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.569 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.569 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:34:32.569 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.569 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.569 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.569 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.569 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:34:32.569 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.569 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.569 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.570 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.570 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:34:32.570 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.570 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.570 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.570 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.570 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:34:32.570 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.570 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.570 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.570 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.570 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:34:32.570 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.571 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.571 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.571 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:34:32.629 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:34:32.630 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:32.630 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-01-23T08:34:32.630 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:32.630 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-01-23T08:34:32.631 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel x86_64 2:17.2.7-397.gb824e5be.el9 ceph 126 k 2024-01-23T08:34:32.631 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:32.631 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-01-23T08:34:32.631 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:32.631 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-01-23T08:34:32.631 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:32.631 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 126 k 2024-01-23T08:34:32.631 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 448 k 2024-01-23T08:34:32.631 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-01-23T08:34:32.860 INFO:teuthology.orchestra.run.smithi137.stdout:librados-devel-17.2.7-397.gb824e5be.el9.x86_64. 553 kB/s | 126 kB 00:00 2024-01-23T08:34:32.860 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:34:32.861 INFO:teuthology.orchestra.run.smithi137.stdout:Total 549 kB/s | 126 kB 00:00 2024-01-23T08:34:32.861 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-01-23T08:34:32.866 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-01-23T08:34:32.866 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-01-23T08:34:32.927 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-01-23T08:34:32.928 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-01-23T08:34:33.075 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-01-23T08:34:33.166 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librados-devel-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:34:33.591 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados-devel-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:34:33.933 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados-devel-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:34:33.933 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:33.933 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-01-23T08:34:33.933 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:34:33.934 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:33.934 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:34:34.170 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs2 2024-01-23T08:34:34.786 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:34 ago on Tue 23 Jan 2024 08:33:00 AM UTC. 2024-01-23T08:34:35.082 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:34:35.083 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:34:35.083 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:34:35.083 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.083 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:34:35.083 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.083 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:34:35.083 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.083 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:34:35.083 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.083 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:34:35.083 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.084 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:34:35.084 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.084 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:34:35.084 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.084 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:34:35.084 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.084 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:34:35.084 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.084 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:34:35.084 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.084 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:34:35.084 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.084 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:34:35.085 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.085 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:34:35.085 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.085 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:34:35.085 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.085 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:34:35.085 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.085 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:34:35.085 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.085 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:34:35.085 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.085 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:34:35.085 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.086 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:34:35.086 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.086 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:34:35.086 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.086 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:34:35.086 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.086 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:34:35.086 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.086 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:34:35.086 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.086 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:34:35.086 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.086 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:34:35.086 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.087 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:34:35.087 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.087 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:34:35.087 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.087 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:34:35.087 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.087 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:34:35.087 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.087 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:34:35.087 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.087 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:34:35.087 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.087 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:34:35.088 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.088 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:34:35.088 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.088 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:34:35.088 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.088 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:34:35.088 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.088 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:34:35.088 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.088 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:34:35.088 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.088 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.088 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.089 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:34:35.089 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.089 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.089 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.089 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.089 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:34:35.089 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.089 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.089 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.089 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.089 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.089 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.089 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:34:35.089 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.089 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.090 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.090 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.090 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:34:35.090 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.090 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.090 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.090 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.090 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:34:35.090 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.090 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.090 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.090 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.090 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:34:35.090 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.090 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.091 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.091 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.091 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:34:35.091 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.091 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.091 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.091 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.091 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:34:35.091 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.091 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.091 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.091 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:34:35.092 INFO:teuthology.orchestra.run.smithi137.stdout:Package libcephfs2-2:17.2.7-397.gb824e5be.el9.x86_64 is already installed. 2024-01-23T08:34:35.149 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:34:35.150 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-01-23T08:34:35.150 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:34:35.225 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs-devel 2024-01-23T08:34:35.905 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:35 ago on Tue 23 Jan 2024 08:33:00 AM UTC. 2024-01-23T08:34:36.194 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:34:36.194 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:34:36.194 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:34:36.194 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.194 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:34:36.195 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.195 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:34:36.195 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.195 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:34:36.195 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.195 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:34:36.195 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.195 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:34:36.195 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.195 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:34:36.195 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.195 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:34:36.195 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.195 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:34:36.196 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.196 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:34:36.196 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.196 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:34:36.196 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.196 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:34:36.196 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.196 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:34:36.196 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.196 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:34:36.196 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.196 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:34:36.196 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.196 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:34:36.197 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.197 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:34:36.197 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.197 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:34:36.197 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.197 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:34:36.197 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.197 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:34:36.197 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.197 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:34:36.197 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.197 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:34:36.197 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.197 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:34:36.198 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.198 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:34:36.198 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.198 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:34:36.198 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.198 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:34:36.198 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.198 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:34:36.198 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.198 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:34:36.198 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.198 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:34:36.198 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.198 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:34:36.199 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.199 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:34:36.199 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.199 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:34:36.199 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.199 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:34:36.199 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.199 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:34:36.199 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.199 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:34:36.199 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.199 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:34:36.199 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.199 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:34:36.200 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.200 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.200 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.200 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:34:36.200 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.200 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.200 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.200 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.200 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:34:36.200 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.200 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.200 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.200 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.200 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.201 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.201 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:34:36.201 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.201 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.201 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.201 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.201 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:34:36.201 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.201 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.201 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.201 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.201 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:34:36.201 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.201 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.202 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.202 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.202 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:34:36.202 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.202 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.202 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.202 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.202 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:34:36.202 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.202 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.202 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.202 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.202 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:34:36.202 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.203 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.203 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.203 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:34:36.262 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:34:36.263 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:36.263 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-01-23T08:34:36.263 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:36.263 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-01-23T08:34:36.263 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel x86_64 2:17.2.7-397.gb824e5be.el9 ceph 25 k 2024-01-23T08:34:36.263 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:36.263 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-01-23T08:34:36.263 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:36.263 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-01-23T08:34:36.263 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:36.264 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 25 k 2024-01-23T08:34:36.264 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 110 k 2024-01-23T08:34:36.264 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-01-23T08:34:36.504 INFO:teuthology.orchestra.run.smithi137.stdout:libcephfs-devel-17.2.7-397.gb824e5be.el9.x86_64 104 kB/s | 25 kB 00:00 2024-01-23T08:34:36.505 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:34:36.505 INFO:teuthology.orchestra.run.smithi137.stdout:Total 103 kB/s | 25 kB 00:00 2024-01-23T08:34:36.505 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-01-23T08:34:36.508 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-01-23T08:34:36.511 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-01-23T08:34:36.512 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-01-23T08:34:36.512 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-01-23T08:34:36.617 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-01-23T08:34:36.752 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs-devel-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:34:37.172 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:34:37.474 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs-devel-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:34:37.475 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:37.475 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-01-23T08:34:37.475 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:34:37.475 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:37.475 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:34:37.659 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados2 2024-01-23T08:34:38.258 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:38 ago on Tue 23 Jan 2024 08:33:00 AM UTC. 2024-01-23T08:34:38.553 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:34:38.554 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:34:38.554 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:34:38.554 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.554 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:34:38.554 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.554 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:34:38.554 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.554 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:34:38.554 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.554 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:34:38.555 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.555 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:34:38.555 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.555 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:34:38.555 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.555 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:34:38.555 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.555 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:34:38.555 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.555 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:34:38.555 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.555 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:34:38.555 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.555 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:34:38.556 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.556 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:34:38.556 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.556 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:34:38.556 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.556 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:34:38.556 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.556 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:34:38.556 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.556 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:34:38.556 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.556 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:34:38.556 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.557 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:34:38.557 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.557 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:34:38.557 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.557 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:34:38.557 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.557 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:34:38.557 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.557 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:34:38.557 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.557 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:34:38.557 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.557 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:34:38.557 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.558 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:34:38.558 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.558 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:34:38.558 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.558 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:34:38.558 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.558 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:34:38.558 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.558 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:34:38.558 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.558 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:34:38.558 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.558 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:34:38.558 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.559 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:34:38.559 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.559 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:34:38.559 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.559 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:34:38.559 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.559 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:34:38.559 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.559 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:34:38.559 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.559 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.559 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.559 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:34:38.559 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.560 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.560 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.560 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.560 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:34:38.560 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.560 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.560 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.560 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.560 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.560 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.560 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:34:38.560 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.560 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.560 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.561 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.561 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:34:38.561 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.561 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.561 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.561 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.561 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:34:38.561 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.561 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.561 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.561 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.561 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:34:38.561 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.561 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.562 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.562 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.562 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:34:38.562 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.562 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.562 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.562 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.562 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:34:38.562 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.562 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.562 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.562 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:34:38.563 INFO:teuthology.orchestra.run.smithi137.stdout:Package librados2-2:17.2.7-397.gb824e5be.el9.x86_64 is already installed. 2024-01-23T08:34:38.620 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:34:38.621 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-01-23T08:34:38.622 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:34:38.696 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librbd1 2024-01-23T08:34:39.295 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:39 ago on Tue 23 Jan 2024 08:33:00 AM UTC. 2024-01-23T08:34:39.586 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:34:39.587 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:34:39.587 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:34:39.587 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.587 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:34:39.587 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.587 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:34:39.587 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.587 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:34:39.587 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.587 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:34:39.587 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.588 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:34:39.588 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.588 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:34:39.588 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.588 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:34:39.588 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.588 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:34:39.588 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.588 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:34:39.588 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.588 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:34:39.588 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.588 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:34:39.589 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.589 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:34:39.589 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.589 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:34:39.589 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.589 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:34:39.589 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.589 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:34:39.589 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.589 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:34:39.589 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.589 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:34:39.589 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.590 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:34:39.590 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.590 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:34:39.590 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.590 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:34:39.590 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.590 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:34:39.590 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.590 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:34:39.590 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.590 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:34:39.590 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.590 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:34:39.591 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.591 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:34:39.591 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.591 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:34:39.591 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.591 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:34:39.591 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.591 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:34:39.591 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.591 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:34:39.591 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.591 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:34:39.591 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.591 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:34:39.592 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.592 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:34:39.592 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.592 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:34:39.592 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.592 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:34:39.592 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.592 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:34:39.592 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.592 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:34:39.592 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.592 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.593 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.593 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:34:39.593 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.593 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.593 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.593 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.593 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:34:39.593 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.593 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.593 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.593 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.593 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.593 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.593 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:34:39.593 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.594 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.594 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.594 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.594 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:34:39.594 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.594 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.594 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.594 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.594 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:34:39.594 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.594 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.594 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.594 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.595 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:34:39.595 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.595 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.595 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.595 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.595 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:34:39.595 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.595 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.595 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.595 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.595 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:34:39.595 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.595 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.595 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.595 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:34:39.596 INFO:teuthology.orchestra.run.smithi137.stdout:Package librbd1-2:17.2.7-397.gb824e5be.el9.x86_64 is already installed. 2024-01-23T08:34:39.654 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:34:39.655 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-01-23T08:34:39.655 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:34:39.730 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rados 2024-01-23T08:34:40.331 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:40 ago on Tue 23 Jan 2024 08:33:00 AM UTC. 2024-01-23T08:34:40.623 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:34:40.623 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:34:40.624 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:34:40.624 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.624 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:34:40.624 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.624 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:34:40.624 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.624 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:34:40.625 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.625 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:34:40.625 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.625 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:34:40.625 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.625 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:34:40.625 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.625 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:34:40.625 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.625 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:34:40.625 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.625 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:34:40.625 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.625 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:34:40.626 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.626 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:34:40.626 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.626 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:34:40.626 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.626 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:34:40.626 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.626 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:34:40.626 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.626 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:34:40.626 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.626 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:34:40.626 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.626 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:34:40.626 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.627 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:34:40.627 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.627 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:34:40.627 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.627 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:34:40.627 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.627 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:34:40.627 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.627 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:34:40.627 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.627 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:34:40.627 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.627 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:34:40.627 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.628 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:34:40.628 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.628 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:34:40.628 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.628 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:34:40.628 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.628 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:34:40.628 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.628 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:34:40.628 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.628 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:34:40.628 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.628 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:34:40.628 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.629 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:34:40.629 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.629 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:34:40.629 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.629 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:34:40.629 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.629 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:34:40.629 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.629 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:34:40.629 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.629 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.629 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.629 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:34:40.629 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.629 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.630 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.630 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.630 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:34:40.630 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.630 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.630 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.630 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.630 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.630 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.630 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:34:40.630 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.630 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.630 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.630 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.630 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:34:40.631 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.631 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.631 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.631 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.631 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:34:40.631 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.631 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.631 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.631 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.631 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:34:40.631 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.631 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.631 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.631 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.632 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:34:40.632 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.632 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.632 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.632 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.632 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:34:40.632 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.632 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.632 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.632 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:34:40.633 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rados-2:17.2.7-397.gb824e5be.el9.x86_64 is already installed. 2024-01-23T08:34:40.690 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:34:40.691 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-01-23T08:34:40.691 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:34:40.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rgw 2024-01-23T08:34:41.361 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:41 ago on Tue 23 Jan 2024 08:33:00 AM UTC. 2024-01-23T08:34:41.652 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:34:41.652 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:34:41.652 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:34:41.652 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.652 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:34:41.652 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.652 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:34:41.652 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.652 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:34:41.652 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.652 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:34:41.653 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.653 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:34:41.653 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.653 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:34:41.653 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.653 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:34:41.653 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.653 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:34:41.653 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.653 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:34:41.653 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.653 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:34:41.653 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.653 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:34:41.654 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.654 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:34:41.654 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.654 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:34:41.654 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.654 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:34:41.654 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.654 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:34:41.654 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.654 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:34:41.654 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.654 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:34:41.654 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.654 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:34:41.655 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.655 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:34:41.655 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.655 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:34:41.655 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.655 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:34:41.655 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.655 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:34:41.655 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.655 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:34:41.655 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.655 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:34:41.655 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.656 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:34:41.656 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.656 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:34:41.656 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.656 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:34:41.656 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.656 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:34:41.656 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.656 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:34:41.656 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.656 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:34:41.656 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.656 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:34:41.656 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.657 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:34:41.657 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.657 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:34:41.657 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.657 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:34:41.657 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.657 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:34:41.657 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.657 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:34:41.657 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.657 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.657 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.657 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:34:41.657 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.657 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.658 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.658 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.658 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:34:41.658 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.658 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.658 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.658 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.658 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.658 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.658 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:34:41.658 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.658 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.658 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.658 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.658 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:34:41.659 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.659 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.659 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.659 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.659 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:34:41.659 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.659 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.659 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.659 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.659 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:34:41.659 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.659 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.659 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.659 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.659 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:34:41.659 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.660 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.660 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.660 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.660 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:34:41.660 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.660 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.660 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.660 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:34:41.660 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rgw-2:17.2.7-397.gb824e5be.el9.x86_64 is already installed. 2024-01-23T08:34:41.719 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:34:41.720 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-01-23T08:34:41.720 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:34:41.798 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-cephfs 2024-01-23T08:34:42.398 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:42 ago on Tue 23 Jan 2024 08:33:00 AM UTC. 2024-01-23T08:34:42.688 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:34:42.688 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:34:42.688 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:34:42.688 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.689 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:34:42.689 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.689 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:34:42.689 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.689 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:34:42.689 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.689 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:34:42.689 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.689 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:34:42.689 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.689 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:34:42.689 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.689 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:34:42.690 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.690 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:34:42.690 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.690 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:34:42.690 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.690 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:34:42.690 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.690 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:34:42.690 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.690 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:34:42.690 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.690 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:34:42.690 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.690 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:34:42.690 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.691 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:34:42.691 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.691 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:34:42.691 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.691 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:34:42.691 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.691 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:34:42.691 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.691 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:34:42.691 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.691 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:34:42.691 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.691 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:34:42.691 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.692 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:34:42.692 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.692 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:34:42.692 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.692 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:34:42.692 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.692 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:34:42.692 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.692 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:34:42.692 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.692 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:34:42.692 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.692 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:34:42.693 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.693 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:34:42.693 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.693 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:34:42.693 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.693 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:34:42.693 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.693 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:34:42.693 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.693 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:34:42.693 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.693 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:34:42.693 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.693 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:34:42.694 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.694 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:34:42.694 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.694 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.694 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.694 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:34:42.694 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.694 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.694 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.694 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.694 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:34:42.694 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.694 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.694 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.695 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.695 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.695 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.695 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:34:42.695 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.695 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.695 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.695 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.695 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:34:42.695 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.695 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.695 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.696 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.696 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:34:42.696 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.696 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.696 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.696 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.696 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:34:42.696 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.696 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.696 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.696 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.697 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:34:42.697 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.697 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.697 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.697 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.697 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:34:42.697 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.697 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.697 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.697 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:34:42.698 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-cephfs-2:17.2.7-397.gb824e5be.el9.x86_64 is already installed. 2024-01-23T08:34:42.755 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:34:42.756 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-01-23T08:34:42.756 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:34:42.827 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rbd 2024-01-23T08:34:43.430 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:43 ago on Tue 23 Jan 2024 08:33:00 AM UTC. 2024-01-23T08:34:43.723 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:34:43.723 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:34:43.723 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:34:43.723 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.724 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:34:43.724 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.724 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:34:43.724 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.724 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:34:43.724 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.724 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:34:43.724 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.724 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:34:43.724 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.724 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:34:43.724 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.725 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:34:43.725 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.725 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:34:43.725 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.725 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:34:43.725 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.725 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:34:43.725 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.725 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:34:43.725 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.725 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:34:43.725 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.725 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:34:43.725 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.725 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:34:43.726 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.726 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:34:43.726 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.726 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:34:43.726 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.726 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:34:43.726 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.726 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:34:43.726 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.726 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:34:43.726 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.726 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:34:43.726 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.726 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:34:43.726 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.727 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:34:43.727 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.727 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:34:43.727 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.727 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:34:43.727 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.727 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:34:43.727 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.727 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:34:43.727 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.727 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:34:43.727 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.727 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:34:43.727 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.728 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:34:43.728 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.728 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:34:43.728 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.728 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:34:43.728 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.728 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:34:43.728 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.728 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:34:43.728 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.728 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:34:43.728 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.728 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:34:43.729 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.729 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:34:43.729 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.729 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.729 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.729 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:34:43.729 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.729 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.729 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.729 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.729 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:34:43.729 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.729 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.729 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.729 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.730 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.730 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.730 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:34:43.730 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.730 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.730 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.730 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.730 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:34:43.730 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.730 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.730 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.730 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.730 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:34:43.731 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.731 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.731 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.731 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.731 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:34:43.731 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.732 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.732 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.732 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.732 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:34:43.732 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.732 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.732 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.732 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.732 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:34:43.732 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.732 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.732 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.732 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:34:43.733 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rbd-2:17.2.7-397.gb824e5be.el9.x86_64 is already installed. 2024-01-23T08:34:43.790 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:34:43.791 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-01-23T08:34:43.791 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:34:43.863 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-fuse 2024-01-23T08:34:44.489 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:44 ago on Tue 23 Jan 2024 08:33:00 AM UTC. 2024-01-23T08:34:44.781 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:34:44.782 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:34:44.782 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:34:44.782 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.782 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:34:44.782 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.782 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:34:44.782 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.782 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:34:44.782 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.782 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:34:44.782 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.782 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:34:44.782 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.783 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:34:44.783 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.783 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:34:44.783 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.783 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:34:44.783 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.783 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:34:44.783 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.783 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:34:44.783 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.783 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:34:44.783 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.783 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:34:44.783 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.783 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:34:44.783 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.784 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:34:44.784 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.784 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:34:44.784 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.784 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:34:44.784 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.784 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:34:44.784 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.784 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:34:44.784 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.784 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:34:44.784 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.784 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:34:44.784 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.785 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:34:44.785 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.785 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:34:44.785 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.785 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:34:44.785 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.785 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:34:44.785 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.785 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:34:44.785 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.785 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:34:44.785 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.785 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:34:44.785 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.785 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:34:44.786 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.786 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:34:44.786 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.786 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:34:44.786 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.786 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:34:44.786 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.786 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:34:44.786 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.786 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:34:44.786 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.786 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:34:44.786 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.786 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:34:44.786 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.787 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:34:44.787 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.787 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.787 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.787 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:34:44.787 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.787 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.787 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.787 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.787 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:34:44.787 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.787 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.787 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.787 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.787 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.787 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.788 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:34:44.788 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.788 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.788 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.788 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.788 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:34:44.788 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.788 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.788 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.788 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.788 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:34:44.788 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.788 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.788 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.789 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.789 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:34:44.789 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.789 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.789 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.789 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.789 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:34:44.789 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.789 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.789 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.789 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.789 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:34:44.789 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.789 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.789 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.789 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:34:44.849 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:34:44.850 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:44.850 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-01-23T08:34:44.850 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:44.850 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-01-23T08:34:44.850 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse x86_64 2:17.2.7-397.gb824e5be.el9 ceph 81 k 2024-01-23T08:34:44.850 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:44.850 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-01-23T08:34:44.850 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:44.850 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-01-23T08:34:44.850 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:44.851 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 81 k 2024-01-23T08:34:44.851 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 222 k 2024-01-23T08:34:44.851 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-01-23T08:34:45.224 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-fuse-17.2.7-397.gb824e5be.el9.x86_64.rpm 218 kB/s | 81 kB 00:00 2024-01-23T08:34:45.225 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:34:45.225 INFO:teuthology.orchestra.run.smithi137.stdout:Total 217 kB/s | 81 kB 00:00 2024-01-23T08:34:45.226 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-01-23T08:34:45.232 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-01-23T08:34:45.232 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-01-23T08:34:45.289 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-01-23T08:34:45.290 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-01-23T08:34:45.425 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-01-23T08:34:45.514 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-fuse-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:34:45.937 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-fuse-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:34:46.284 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-fuse-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:34:46.284 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:46.284 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-01-23T08:34:46.284 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:34:46.284 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:46.284 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:34:46.516 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-mirror 2024-01-23T08:34:47.125 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:47 ago on Tue 23 Jan 2024 08:33:00 AM UTC. 2024-01-23T08:34:47.420 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:34:47.420 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:34:47.421 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:34:47.421 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.421 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:34:47.421 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.421 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:34:47.421 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.421 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:34:47.421 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.421 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:34:47.421 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.421 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:34:47.421 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.421 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:34:47.421 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.422 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:34:47.422 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.422 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:34:47.422 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.422 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:34:47.422 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.422 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:34:47.422 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.422 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:34:47.422 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.422 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:34:47.422 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.422 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:34:47.422 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.422 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:34:47.423 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.423 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:34:47.423 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.423 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:34:47.423 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.423 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:34:47.423 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.423 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:34:47.423 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.423 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:34:47.423 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.423 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:34:47.423 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.423 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:34:47.424 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.424 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:34:47.424 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.424 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:34:47.424 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.424 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:34:47.424 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.424 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:34:47.424 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.424 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:34:47.424 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.424 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:34:47.424 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.424 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:34:47.425 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.425 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:34:47.425 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.425 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:34:47.425 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.425 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:34:47.425 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.425 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:34:47.425 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.425 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:34:47.425 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.425 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:34:47.426 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.426 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:34:47.426 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.426 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:34:47.426 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.426 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.426 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.426 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:34:47.426 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.426 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.426 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.426 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.426 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:34:47.426 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.427 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.427 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.427 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.427 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.427 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.427 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:34:47.427 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.427 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.427 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.427 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.427 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:34:47.427 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.427 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.427 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.427 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.428 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:34:47.428 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.428 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.428 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.428 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.428 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:34:47.428 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.428 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.428 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.428 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.428 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:34:47.428 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.428 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.429 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.429 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.429 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:34:47.429 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.429 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.429 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.429 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:34:47.488 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:34:47.488 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:47.488 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-01-23T08:34:47.488 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:47.488 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-01-23T08:34:47.488 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror x86_64 2:17.2.7-397.gb824e5be.el9 ceph 2.9 M 2024-01-23T08:34:47.488 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:47.488 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-01-23T08:34:47.489 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:47.489 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-01-23T08:34:47.489 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:47.489 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 2.9 M 2024-01-23T08:34:47.489 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 12 M 2024-01-23T08:34:47.489 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-01-23T08:34:48.260 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-mirror-17.2.7-397.gb824e5be.el9.x86_64.rpm 3.8 MB/s | 2.9 MB 00:00 2024-01-23T08:34:48.260 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:34:48.261 INFO:teuthology.orchestra.run.smithi137.stdout:Total 3.8 MB/s | 2.9 MB 00:00 2024-01-23T08:34:48.261 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-01-23T08:34:48.270 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-01-23T08:34:48.270 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-01-23T08:34:48.345 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-01-23T08:34:48.345 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-01-23T08:34:48.762 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-01-23T08:34:48.852 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-mirror-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:34:48.877 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-mirror-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:34:48.877 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-23T08:34:48.877 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-23T08:34:48.877 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-23T08:34:48.877 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-23T08:34:48.877 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:50.053 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-mirror-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:34:50.053 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:50.053 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-01-23T08:34:50.053 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:34:50.053 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:50.053 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:34:50.302 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-nbd 2024-01-23T08:34:50.919 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:50 ago on Tue 23 Jan 2024 08:33:00 AM UTC. 2024-01-23T08:34:51.213 INFO:teuthology.orchestra.run.smithi137.stderr:Modular dependency problems: 2024-01-23T08:34:51.213 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-23T08:34:51.213 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 1: conflicting requests 2024-01-23T08:34:51.213 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.213 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 2: conflicting requests 2024-01-23T08:34:51.213 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.213 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 3: conflicting requests 2024-01-23T08:34:51.213 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.213 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 4: conflicting requests 2024-01-23T08:34:51.214 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.214 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 5: conflicting requests 2024-01-23T08:34:51.214 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.214 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 6: conflicting requests 2024-01-23T08:34:51.214 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.214 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 7: conflicting requests 2024-01-23T08:34:51.214 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.214 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 8: conflicting requests 2024-01-23T08:34:51.214 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.214 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 9: conflicting requests 2024-01-23T08:34:51.214 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.214 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 10: conflicting requests 2024-01-23T08:34:51.214 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.214 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 11: conflicting requests 2024-01-23T08:34:51.214 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.215 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 12: conflicting requests 2024-01-23T08:34:51.215 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.215 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 13: conflicting requests 2024-01-23T08:34:51.215 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.215 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 14: conflicting requests 2024-01-23T08:34:51.215 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.215 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 15: conflicting requests 2024-01-23T08:34:51.215 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.215 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 16: conflicting requests 2024-01-23T08:34:51.215 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.215 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 17: conflicting requests 2024-01-23T08:34:51.215 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.215 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 18: conflicting requests 2024-01-23T08:34:51.215 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.215 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 19: conflicting requests 2024-01-23T08:34:51.216 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.216 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 20: conflicting requests 2024-01-23T08:34:51.216 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.216 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 21: conflicting requests 2024-01-23T08:34:51.216 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.216 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 22: conflicting requests 2024-01-23T08:34:51.216 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.216 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 23: conflicting requests 2024-01-23T08:34:51.216 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.216 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 24: conflicting requests 2024-01-23T08:34:51.216 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.217 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 25: conflicting requests 2024-01-23T08:34:51.217 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.217 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 26: conflicting requests 2024-01-23T08:34:51.217 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.217 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 27: conflicting requests 2024-01-23T08:34:51.217 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.217 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 28: conflicting requests 2024-01-23T08:34:51.217 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.217 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 29: conflicting requests 2024-01-23T08:34:51.217 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.217 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 30: conflicting requests 2024-01-23T08:34:51.217 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.217 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 31: conflicting requests 2024-01-23T08:34:51.218 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.218 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 32: conflicting requests 2024-01-23T08:34:51.218 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.218 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 33: conflicting requests 2024-01-23T08:34:51.218 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.218 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 34: conflicting requests 2024-01-23T08:34:51.218 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.218 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 35: conflicting requests 2024-01-23T08:34:51.218 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.218 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 36: conflicting requests 2024-01-23T08:34:51.218 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.218 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 37: conflicting requests 2024-01-23T08:34:51.218 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.219 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.219 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.219 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:34:51.219 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.219 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.219 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.219 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.219 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 39: conflicting requests 2024-01-23T08:34:51.219 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.219 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.219 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.219 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.219 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.219 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.220 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:34:51.220 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.220 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.220 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.220 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.220 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:34:51.220 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.220 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.220 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.220 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.220 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:34:51.221 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.221 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.221 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.221 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.221 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:34:51.221 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.221 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.221 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.221 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.221 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:34:51.221 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.221 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.221 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.221 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.222 INFO:teuthology.orchestra.run.smithi137.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:34:51.222 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.222 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.222 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.222 INFO:teuthology.orchestra.run.smithi137.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:34:51.281 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-23T08:34:51.282 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:51.282 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-01-23T08:34:51.282 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:51.282 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-01-23T08:34:51.282 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd x86_64 2:17.2.7-397.gb824e5be.el9 ceph 164 k 2024-01-23T08:34:51.282 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:51.283 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-01-23T08:34:51.283 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-23T08:34:51.283 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-01-23T08:34:51.283 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:51.283 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 164 k 2024-01-23T08:34:51.283 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 477 k 2024-01-23T08:34:51.283 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-01-23T08:34:51.514 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-nbd-17.2.7-397.gb824e5be.el9.x86_64.rpm 713 kB/s | 164 kB 00:00 2024-01-23T08:34:51.514 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:34:51.514 INFO:teuthology.orchestra.run.smithi137.stdout:Total 708 kB/s | 164 kB 00:00 2024-01-23T08:34:51.514 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-01-23T08:34:51.522 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-01-23T08:34:51.522 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-01-23T08:34:51.580 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-01-23T08:34:51.581 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-01-23T08:34:51.725 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-01-23T08:34:51.835 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-nbd-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:34:52.244 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-nbd-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:34:52.570 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-nbd-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:34:52.570 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:52.570 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-01-23T08:34:52.571 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:34:52.571 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:34:52.571 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-23T08:34:52.784 DEBUG:teuthology.parallel:result is None 2024-01-23T08:37:37.130 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 78 kB/s | 20 MB 04:24 2024-01-23T08:37:42.177 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-01-23T08:37:44.770 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:37:44.770 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:37:44.770 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:37:44.770 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.770 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:37:44.770 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.770 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:37:44.771 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.771 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:37:44.771 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.771 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:37:44.771 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.771 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:37:44.771 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.771 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:37:44.771 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.771 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:37:44.771 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.771 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:37:44.772 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.772 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:37:44.772 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.772 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:37:44.772 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.772 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:37:44.772 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.772 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:37:44.772 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.772 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:37:44.772 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.772 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:37:44.773 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.773 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:37:44.773 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.773 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:37:44.773 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.773 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:37:44.773 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.773 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:37:44.773 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.773 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:37:44.773 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.773 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:37:44.774 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.774 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:37:44.774 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.774 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:37:44.774 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.774 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:37:44.774 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.774 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:37:44.774 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.774 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:37:44.774 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.774 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:37:44.774 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.774 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:37:44.775 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.775 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:37:44.775 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.775 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:37:44.775 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.775 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:37:44.775 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.775 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:37:44.775 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.775 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:37:44.775 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.775 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:37:44.775 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.776 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:37:44.776 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.776 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:37:44.776 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.776 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:37:44.776 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.776 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.776 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.776 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:37:44.776 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.776 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.776 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.776 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.777 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:37:44.777 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.777 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.777 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.777 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.777 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.777 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.777 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:37:44.777 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.777 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.777 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.777 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.777 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:37:44.777 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.778 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.778 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.778 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.778 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:37:44.778 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.778 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.778 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.778 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.778 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:37:44.778 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.778 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.778 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.778 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.778 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:37:44.779 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.779 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.779 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.779 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.779 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:37:44.779 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.779 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.779 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:37:44.779 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:37:45.260 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:37:45.270 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:37:45.270 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-01-23T08:37:45.270 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:37:45.270 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-23T08:37:45.270 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw x86_64 2:17.2.7-397.gb824e5be.el9 ceph 14 M 2024-01-23T08:37:45.270 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrading: 2024-01-23T08:37:45.270 INFO:teuthology.orchestra.run.smithi059.stdout: librados2 x86_64 2:17.2.7-397.gb824e5be.el9 ceph 3.1 M 2024-01-23T08:37:45.270 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1 x86_64 2:17.2.7-397.gb824e5be.el9 ceph 2.9 M 2024-01-23T08:37:45.271 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-01-23T08:37:45.271 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base x86_64 2:17.2.7-397.gb824e5be.el9 ceph 5.0 M 2024-01-23T08:37:45.271 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common x86_64 2:17.2.7-397.gb824e5be.el9 ceph 23 M 2024-01-23T08:37:45.271 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux x86_64 2:17.2.7-397.gb824e5be.el9 ceph 25 k 2024-01-23T08:37:45.271 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-23T08:37:45.271 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2 x86_64 2:17.2.7-397.gb824e5be.el9 ceph 635 k 2024-01-23T08:37:45.271 INFO:teuthology.orchestra.run.smithi059.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-23T08:37:45.271 INFO:teuthology.orchestra.run.smithi059.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-01-23T08:37:45.271 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-01-23T08:37:45.271 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1 x86_64 2:17.2.7-397.gb824e5be.el9 ceph 459 k 2024-01-23T08:37:45.271 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-01-23T08:37:45.271 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2 x86_64 2:17.2.7-397.gb824e5be.el9 ceph 4.9 M 2024-01-23T08:37:45.271 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-23T08:37:45.272 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-01-23T08:37:45.272 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse x86_64 2:17.2.7-397.gb824e5be.el9 ceph 45 k 2024-01-23T08:37:45.272 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common x86_64 2:17.2.7-397.gb824e5be.el9 ceph 116 k 2024-01-23T08:37:45.272 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs x86_64 2:17.2.7-397.gb824e5be.el9 ceph 156 k 2024-01-23T08:37:45.272 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-01-23T08:37:45.272 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados x86_64 2:17.2.7-397.gb824e5be.el9 ceph 320 k 2024-01-23T08:37:45.272 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd x86_64 2:17.2.7-397.gb824e5be.el9 ceph 293 k 2024-01-23T08:37:45.272 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw x86_64 2:17.2.7-397.gb824e5be.el9 ceph 99 k 2024-01-23T08:37:45.272 INFO:teuthology.orchestra.run.smithi059.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-23T08:37:45.272 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:37:45.272 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-23T08:37:45.272 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:37:45.272 INFO:teuthology.orchestra.run.smithi059.stdout:Install 22 Packages 2024-01-23T08:37:45.273 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrade 2 Packages 2024-01-23T08:37:45.273 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:37:45.273 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 58 M 2024-01-23T08:37:45.273 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-23T08:37:46.236 INFO:teuthology.orchestra.run.smithi059.stdout:(1/24): ceph-base-17.2.7-397.gb824e5be.el9.x86_ 7.4 MB/s | 5.0 MB 00:00 2024-01-23T08:37:46.294 INFO:teuthology.orchestra.run.smithi059.stdout:(2/24): ceph-selinux-17.2.7-397.gb824e5be.el9.x 432 kB/s | 25 kB 00:00 2024-01-23T08:37:46.403 INFO:teuthology.orchestra.run.smithi059.stdout:(3/24): libcephfs2-17.2.7-397.gb824e5be.el9.x86 5.7 MB/s | 635 kB 00:00 2024-01-23T08:37:46.487 INFO:teuthology.orchestra.run.smithi059.stdout:(4/24): libradosstriper1-17.2.7-397.gb824e5be.e 5.4 MB/s | 459 kB 00:00 2024-01-23T08:37:47.145 INFO:teuthology.orchestra.run.smithi059.stdout:(5/24): librgw2-17.2.7-397.gb824e5be.el9.x86_64 7.4 MB/s | 4.9 MB 00:00 2024-01-23T08:37:47.221 INFO:teuthology.orchestra.run.smithi059.stdout:(6/24): python3-ceph-argparse-17.2.7-397.gb824e 603 kB/s | 45 kB 00:00 2024-01-23T08:37:47.446 INFO:teuthology.orchestra.run.smithi059.stdout:(7/24): ceph-common-17.2.7-397.gb824e5be.el9.x8 12 MB/s | 23 MB 00:01 2024-01-23T08:37:47.471 INFO:teuthology.orchestra.run.smithi059.stdout:(8/24): python3-ceph-common-17.2.7-397.gb824e5b 464 kB/s | 116 kB 00:00 2024-01-23T08:37:47.496 INFO:teuthology.orchestra.run.smithi059.stdout:(9/24): python3-cephfs-17.2.7-397.gb824e5be.el9 3.1 MB/s | 156 kB 00:00 2024-01-23T08:37:47.540 INFO:teuthology.orchestra.run.smithi059.stdout:(10/24): python3-rados-17.2.7-397.gb824e5be.el9 4.7 MB/s | 320 kB 00:00 2024-01-23T08:37:47.563 INFO:teuthology.orchestra.run.smithi059.stdout:(11/24): python3-rbd-17.2.7-397.gb824e5be.el9.x 4.3 MB/s | 293 kB 00:00 2024-01-23T08:37:47.589 INFO:teuthology.orchestra.run.smithi059.stdout:(12/24): python3-rgw-17.2.7-397.gb824e5be.el9.x 1.9 MB/s | 99 kB 00:00 2024-01-23T08:37:48.014 INFO:teuthology.orchestra.run.smithi059.stdout:(13/24): ceph-radosgw-17.2.7-397.gb824e5be.el9. 5.7 MB/s | 14 MB 00:02 2024-01-23T08:37:48.039 INFO:teuthology.orchestra.run.smithi059.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 100 kB/s | 45 kB 00:00 2024-01-23T08:37:48.081 INFO:teuthology.orchestra.run.smithi059.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 310 kB/s | 160 kB 00:00 2024-01-23T08:37:48.190 INFO:teuthology.orchestra.run.smithi059.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 390 kB/s | 42 kB 00:00 2024-01-23T08:37:48.440 INFO:teuthology.orchestra.run.smithi059.stdout:(17/24): librdkafka-1.6.1-102.el9.x86_64.rpm 1.5 MB/s | 662 kB 00:00 2024-01-23T08:37:48.624 INFO:teuthology.orchestra.run.smithi059.stdout:(18/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 501 kB/s | 292 kB 00:00 2024-01-23T08:37:48.916 INFO:teuthology.orchestra.run.smithi059.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 108 kB/s | 49 kB 00:00 2024-01-23T08:37:48.999 INFO:teuthology.orchestra.run.smithi059.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 179 kB/s | 67 kB 00:00 2024-01-23T08:37:49.050 INFO:teuthology.orchestra.run.smithi059.stdout:(21/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 359 kB/s | 309 kB 00:00 2024-01-23T08:37:49.275 INFO:teuthology.orchestra.run.smithi059.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 4.4 MB/s | 1.6 MB 00:00 2024-01-23T08:37:49.358 INFO:teuthology.orchestra.run.smithi059.stdout:(23/24): librbd1-17.2.7-397.gb824e5be.el9.x86_6 9.5 MB/s | 2.9 MB 00:00 2024-01-23T08:37:49.434 INFO:teuthology.orchestra.run.smithi059.stdout:(24/24): librados2-17.2.7-397.gb824e5be.el9.x86 7.1 MB/s | 3.1 MB 00:00 2024-01-23T08:37:49.438 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:37:49.439 INFO:teuthology.orchestra.run.smithi059.stdout:Total 14 MB/s | 58 MB 00:04 2024-01-23T08:37:49.497 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-23T08:37:49.533 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-23T08:37:49.533 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-23T08:37:50.024 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-23T08:37:50.025 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-23T08:37:51.282 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-23T08:37:51.627 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-01-23T08:37:51.669 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librados2-2:17.2.7-397.gb824e5be.el9.x86_64 2/26 2024-01-23T08:37:51.764 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:17.2.7-397.gb824e5be.el9.x86_64 2/26 2024-01-23T08:37:51.850 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rados-2:17.2.7-397.gb824e5be.el9.x86_64 3/26 2024-01-23T08:37:51.879 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs2-2:17.2.7-397.gb824e5be.el9.x86_64 4/26 2024-01-23T08:37:51.967 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs2-2:17.2.7-397.gb824e5be.el9.x86_64 4/26 2024-01-23T08:37:52.013 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-01-23T08:37:52.042 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-01-23T08:37:52.488 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-01-23T08:37:52.523 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librgw2-2:17.2.7-397.gb824e5be.el9.x86_64 8/26 2024-01-23T08:37:52.578 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librgw2-2:17.2.7-397.gb824e5be.el9.x86_64 8/26 2024-01-23T08:37:52.615 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-01-23T08:37:52.658 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-argparse-2:17.2.7-397.gb824e5be.el9.x 10/26 2024-01-23T08:37:52.696 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cephfs-2:17.2.7-397.gb824e5be.el9.x86_64 11/26 2024-01-23T08:37:52.759 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rgw-2:17.2.7-397.gb824e5be.el9.x86_64 12/26 2024-01-23T08:37:52.784 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libradosstriper1-2:17.2.7-397.gb824e5be.el9.x86_64 13/26 2024-01-23T08:37:52.833 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libradosstriper1-2:17.2.7-397.gb824e5be.el9.x86_64 13/26 2024-01-23T08:37:52.888 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-01-23T08:37:52.927 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-01-23T08:37:52.959 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-01-23T08:37:53.272 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-01-23T08:37:53.305 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librbd1-2:17.2.7-397.gb824e5be.el9.x86_64 18/26 2024-01-23T08:37:53.395 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:17.2.7-397.gb824e5be.el9.x86_64 18/26 2024-01-23T08:37:53.471 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rbd-2:17.2.7-397.gb824e5be.el9.x86_64 19/26 2024-01-23T08:37:53.498 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-common-2:17.2.7-397.gb824e5be.el9.x86 20/26 2024-01-23T08:37:56.029 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:17.2.7-397.gb824e5be.el9.x86_64 21/26 2024-01-23T08:37:56.062 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-common-2:17.2.7-397.gb824e5be.el9.x86_64 21/26 2024-01-23T08:37:56.624 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:17.2.7-397.gb824e5be.el9.x86_64 21/26 2024-01-23T08:37:56.657 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-base-2:17.2.7-397.gb824e5be.el9.x86_64 22/26 2024-01-23T08:37:56.763 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-base-2:17.2.7-397.gb824e5be.el9.x86_64 22/26 2024-01-23T08:37:56.763 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-23T08:37:56.763 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-23T08:37:56.763 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:37:56.802 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-selinux-2:17.2.7-397.gb824e5be.el9.x86_64 23/26 2024-01-23T08:38:06.648 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-selinux-2:17.2.7-397.gb824e5be.el9.x86_64 23/26 2024-01-23T08:38:06.648 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /sys 2024-01-23T08:38:06.648 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /proc 2024-01-23T08:38:06.649 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /mnt 2024-01-23T08:38:06.649 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /var/tmp 2024-01-23T08:38:06.649 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /home 2024-01-23T08:38:06.649 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /root 2024-01-23T08:38:06.649 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /tmp 2024-01-23T08:38:06.649 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:38:07.960 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-radosgw-2:17.2.7-397.gb824e5be.el9.x86_64 24/26 2024-01-23T08:38:08.021 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-397.gb824e5be.el9.x86_64 24/26 2024-01-23T08:38:08.021 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-23T08:38:08.021 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-23T08:38:08.021 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-23T08:38:08.021 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-23T08:38:08.021 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:38:08.023 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-01-23T08:38:08.096 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-01-23T08:38:08.097 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-01-23T08:38:09.370 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-01-23T08:38:09.370 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-base-2:17.2.7-397.gb824e5be.el9.x86_64 1/26 2024-01-23T08:38:09.370 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-common-2:17.2.7-397.gb824e5be.el9.x86_64 2/26 2024-01-23T08:38:09.370 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-radosgw-2:17.2.7-397.gb824e5be.el9.x86_64 3/26 2024-01-23T08:38:09.370 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-selinux-2:17.2.7-397.gb824e5be.el9.x86_64 4/26 2024-01-23T08:38:09.370 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs2-2:17.2.7-397.gb824e5be.el9.x86_64 5/26 2024-01-23T08:38:09.370 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libradosstriper1-2:17.2.7-397.gb824e5be.el9.x86_64 6/26 2024-01-23T08:38:09.370 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librgw2-2:17.2.7-397.gb824e5be.el9.x86_64 7/26 2024-01-23T08:38:09.370 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-argparse-2:17.2.7-397.gb824e5be.el9.x 8/26 2024-01-23T08:38:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-common-2:17.2.7-397.gb824e5be.el9.x86 9/26 2024-01-23T08:38:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cephfs-2:17.2.7-397.gb824e5be.el9.x86_64 10/26 2024-01-23T08:38:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rados-2:17.2.7-397.gb824e5be.el9.x86_64 11/26 2024-01-23T08:38:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rbd-2:17.2.7-397.gb824e5be.el9.x86_64 12/26 2024-01-23T08:38:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rgw-2:17.2.7-397.gb824e5be.el9.x86_64 13/26 2024-01-23T08:38:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-01-23T08:38:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-01-23T08:38:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-01-23T08:38:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-01-23T08:38:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-01-23T08:38:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-01-23T08:38:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-01-23T08:38:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-01-23T08:38:09.371 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-01-23T08:38:09.372 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:17.2.7-397.gb824e5be.el9.x86_64 23/26 2024-01-23T08:38:09.372 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-01-23T08:38:09.372 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:17.2.7-397.gb824e5be.el9.x86_64 25/26 2024-01-23T08:38:10.173 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-01-23T08:38:10.173 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:38:10.173 INFO:teuthology.orchestra.run.smithi059.stdout:Upgraded: 2024-01-23T08:38:10.173 INFO:teuthology.orchestra.run.smithi059.stdout: librados2-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:38:10.173 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:38:10.173 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-23T08:38:10.173 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:38:10.173 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:38:10.174 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:38:10.174 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:38:10.174 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-23T08:38:10.174 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:38:10.174 INFO:teuthology.orchestra.run.smithi059.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-23T08:38:10.174 INFO:teuthology.orchestra.run.smithi059.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-23T08:38:10.174 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-23T08:38:10.174 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:38:10.174 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-23T08:38:10.174 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:38:10.174 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-23T08:38:10.174 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-23T08:38:10.174 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:38:10.174 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:38:10.174 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:38:10.175 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-23T08:38:10.175 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:38:10.175 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:38:10.175 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:38:10.175 INFO:teuthology.orchestra.run.smithi059.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-23T08:38:10.175 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:38:10.175 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:38:10.465 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-test 2024-01-23T08:38:11.068 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:00:29 ago on Tue 23 Jan 2024 08:37:42 AM UTC. 2024-01-23T08:38:11.375 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:38:11.375 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:38:11.375 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:38:11.375 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.375 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:38:11.375 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.376 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:38:11.376 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.376 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:38:11.376 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.376 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:38:11.376 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.376 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:38:11.376 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.376 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:38:11.376 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.376 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:38:11.376 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.376 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:38:11.377 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.377 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:38:11.377 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.377 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:38:11.377 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.377 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:38:11.377 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.377 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:38:11.377 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.377 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:38:11.377 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.377 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:38:11.377 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.377 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:38:11.378 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.378 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:38:11.378 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.378 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:38:11.378 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.378 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:38:11.378 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.378 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:38:11.378 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.378 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:38:11.378 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.378 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:38:11.378 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.379 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:38:11.379 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.379 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:38:11.379 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.379 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:38:11.379 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.379 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:38:11.379 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.379 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:38:11.379 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.379 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:38:11.379 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.379 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:38:11.379 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.380 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:38:11.380 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.380 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:38:11.380 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.380 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:38:11.380 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.380 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:38:11.380 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.380 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:38:11.380 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.380 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:38:11.380 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.380 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:38:11.381 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.381 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:38:11.381 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.381 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.381 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.381 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:38:11.381 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.381 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.381 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.381 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.381 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:38:11.381 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.381 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.381 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.381 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.382 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.382 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.382 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:38:11.382 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.382 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.382 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.382 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.382 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:38:11.382 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.382 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.382 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.382 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.382 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:38:11.383 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.383 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.383 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.383 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.383 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:38:11.383 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.383 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.383 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.383 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.383 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:38:11.383 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.383 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.383 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.383 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.383 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:38:11.384 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.384 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.384 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.384 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:38:11.457 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:38:11.457 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:38:11.457 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-01-23T08:38:11.458 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:38:11.458 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-23T08:38:11.458 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test x86_64 2:17.2.7-397.gb824e5be.el9 ceph 48 M 2024-01-23T08:38:11.458 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-01-23T08:38:11.458 INFO:teuthology.orchestra.run.smithi059.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-01-23T08:38:11.458 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite x86_64 2:17.2.7-397.gb824e5be.el9 ceph 162 k 2024-01-23T08:38:11.458 INFO:teuthology.orchestra.run.smithi059.stdout: oniguruma x86_64 6.9.6-1.el9.5 baseos 218 k 2024-01-23T08:38:11.458 INFO:teuthology.orchestra.run.smithi059.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-01-23T08:38:11.458 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-01-23T08:38:11.458 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:38:11.458 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-23T08:38:11.458 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:38:11.458 INFO:teuthology.orchestra.run.smithi059.stdout:Install 6 Packages 2024-01-23T08:38:11.459 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:38:11.459 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 49 M 2024-01-23T08:38:11.459 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 195 M 2024-01-23T08:38:11.459 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-23T08:38:11.833 INFO:teuthology.orchestra.run.smithi059.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-01-23T08:38:11.892 INFO:teuthology.orchestra.run.smithi059.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.7 MB/s | 218 kB 00:00 2024-01-23T08:38:11.958 INFO:teuthology.orchestra.run.smithi059.stdout:(3/6): libcephsqlite-17.2.7-397.gb824e5be.el9.x 609 kB/s | 162 kB 00:00 2024-01-23T08:38:12.275 INFO:teuthology.orchestra.run.smithi059.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 201 kB/s | 64 kB 00:00 2024-01-23T08:38:12.301 INFO:teuthology.orchestra.run.smithi059.stdout:(5/6): socat-1.7.4.1-5.el9.x86_64.rpm 745 kB/s | 305 kB 00:00 2024-01-23T08:38:14.351 INFO:teuthology.orchestra.run.smithi059.stdout:(6/6): ceph-test-17.2.7-397.gb824e5be.el9.x86_6 18 MB/s | 48 MB 00:02 2024-01-23T08:38:14.353 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:38:14.353 INFO:teuthology.orchestra.run.smithi059.stdout:Total 17 MB/s | 49 MB 00:02 2024-01-23T08:38:14.403 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-23T08:38:14.424 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-23T08:38:14.424 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-23T08:38:14.838 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-23T08:38:14.838 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-23T08:38:15.445 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-23T08:38:15.521 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-01-23T08:38:15.570 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-01-23T08:38:15.618 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-01-23T08:38:15.659 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-01-23T08:38:15.687 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephsqlite-2:17.2.7-397.gb824e5be.el9.x86_64 5/6 2024-01-23T08:38:20.354 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephsqlite-2:17.2.7-397.gb824e5be.el9.x86_64 5/6 2024-01-23T08:38:20.431 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-test-2:17.2.7-397.gb824e5be.el9.x86_64 6/6 2024-01-23T08:38:22.656 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-test-2:17.2.7-397.gb824e5be.el9.x86_64 6/6 2024-01-23T08:38:22.657 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-test-2:17.2.7-397.gb824e5be.el9.x86_64 1/6 2024-01-23T08:38:22.657 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephsqlite-2:17.2.7-397.gb824e5be.el9.x86_64 2/6 2024-01-23T08:38:22.657 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2024-01-23T08:38:22.657 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2024-01-23T08:38:22.657 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2024-01-23T08:38:23.096 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2024-01-23T08:38:23.097 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:38:23.097 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-23T08:38:23.097 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:38:23.097 INFO:teuthology.orchestra.run.smithi059.stdout: jq-1.6-16.el9.x86_64 2024-01-23T08:38:23.097 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:38:23.097 INFO:teuthology.orchestra.run.smithi059.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-23T08:38:23.097 INFO:teuthology.orchestra.run.smithi059.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-23T08:38:23.097 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-23T08:38:23.097 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:38:23.097 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:38:23.371 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph 2024-01-23T08:38:23.973 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:00:41 ago on Tue 23 Jan 2024 08:37:42 AM UTC. 2024-01-23T08:38:24.278 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:38:24.279 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:38:24.279 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:38:24.279 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.279 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:38:24.279 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.279 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:38:24.279 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.279 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:38:24.279 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.279 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:38:24.279 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.279 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:38:24.280 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.280 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:38:24.280 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.280 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:38:24.280 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.280 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:38:24.280 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.280 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:38:24.280 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.280 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:38:24.280 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.280 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:38:24.280 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.281 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:38:24.281 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.281 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:38:24.281 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.281 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:38:24.281 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.281 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:38:24.281 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.281 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:38:24.281 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.281 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:38:24.281 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.281 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:38:24.281 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.282 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:38:24.282 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.282 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:38:24.282 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.282 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:38:24.282 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.282 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:38:24.282 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.282 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:38:24.282 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.282 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:38:24.282 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.282 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:38:24.282 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.283 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:38:24.283 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.283 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:38:24.283 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.283 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:38:24.283 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.283 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:38:24.283 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.283 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:38:24.283 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.283 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:38:24.283 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.283 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:38:24.284 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.284 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:38:24.284 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.284 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:38:24.284 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.284 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:38:24.284 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.284 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:38:24.284 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.284 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.284 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.284 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:38:24.284 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.284 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.285 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.285 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.285 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:38:24.285 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.285 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.285 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.285 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.285 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.285 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.285 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:38:24.285 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.285 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.285 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.286 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.286 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:38:24.286 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.286 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.286 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.286 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.286 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:38:24.286 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.286 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.286 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.286 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.286 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:38:24.286 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.286 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.286 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.287 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.287 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:38:24.287 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.287 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.287 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.287 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.287 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:38:24.287 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.287 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.287 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.287 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:38:24.352 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:38:24.354 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:38:24.355 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-01-23T08:38:24.355 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:38:24.355 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-23T08:38:24.355 INFO:teuthology.orchestra.run.smithi059.stdout: ceph x86_64 2:17.2.7-397.gb824e5be.el9 ceph 6.5 k 2024-01-23T08:38:24.355 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-01-23T08:38:24.355 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds x86_64 2:17.2.7-397.gb824e5be.el9 ceph 2.1 M 2024-01-23T08:38:24.355 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr x86_64 2:17.2.7-397.gb824e5be.el9 ceph 1.4 M 2024-01-23T08:38:24.355 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core noarch 2:17.2.7-397.gb824e5be.el9 ceph-noarch 239 k 2024-01-23T08:38:24.355 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon x86_64 2:17.2.7-397.gb824e5be.el9 ceph 3.7 M 2024-01-23T08:38:24.355 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd x86_64 2:17.2.7-397.gb824e5be.el9 ceph 15 M 2024-01-23T08:38:24.355 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-23T08:38:24.355 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-01-23T08:38:24.355 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-23T08:38:24.355 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-23T08:38:24.356 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-01-23T08:38:24.356 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-23T08:38:24.356 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-23T08:38:24.356 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-23T08:38:24.356 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-23T08:38:24.356 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-23T08:38:24.356 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-23T08:38:24.356 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-01-23T08:38:24.356 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-01-23T08:38:24.356 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-23T08:38:24.356 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-23T08:38:24.356 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-01-23T08:38:24.356 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-23T08:38:24.357 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-23T08:38:24.357 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-01-23T08:38:24.357 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-01-23T08:38:24.357 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests noarch 2.25.1-7.el9 baseos 124 k 2024-01-23T08:38:24.357 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-23T08:38:24.357 INFO:teuthology.orchestra.run.smithi059.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-01-23T08:38:24.357 INFO:teuthology.orchestra.run.smithi059.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-01-23T08:38:24.357 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-23T08:38:24.357 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-23T08:38:24.357 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-23T08:38:24.357 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:38:24.357 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-23T08:38:24.357 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:38:24.357 INFO:teuthology.orchestra.run.smithi059.stdout:Install 33 Packages 2024-01-23T08:38:24.357 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:38:24.358 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 27 M 2024-01-23T08:38:24.358 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 99 M 2024-01-23T08:38:24.358 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-23T08:38:25.011 INFO:teuthology.orchestra.run.smithi059.stdout:(1/33): ceph-17.2.7-397.gb824e5be.el9.x86_64.rp 31 kB/s | 6.5 kB 00:00 2024-01-23T08:38:25.319 INFO:teuthology.orchestra.run.smithi059.stdout:(2/33): ceph-mds-17.2.7-397.gb824e5be.el9.x86_6 4.0 MB/s | 2.1 MB 00:00 2024-01-23T08:38:25.353 INFO:teuthology.orchestra.run.smithi059.stdout:(3/33): ceph-mgr-17.2.7-397.gb824e5be.el9.x86_6 2.6 MB/s | 1.4 MB 00:00 2024-01-23T08:38:25.453 INFO:teuthology.orchestra.run.smithi059.stdout:(4/33): ceph-mgr-modules-core-17.2.7-397.gb824e 2.3 MB/s | 239 kB 00:00 2024-01-23T08:38:25.595 INFO:teuthology.orchestra.run.smithi059.stdout:(5/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-01-23T08:38:25.687 INFO:teuthology.orchestra.run.smithi059.stdout:(6/33): python3-cryptography-36.0.1-4.el9.x86_6 13 MB/s | 1.2 MB 00:00 2024-01-23T08:38:25.729 INFO:teuthology.orchestra.run.smithi059.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-01-23T08:38:25.771 INFO:teuthology.orchestra.run.smithi059.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-01-23T08:38:25.821 INFO:teuthology.orchestra.run.smithi059.stdout:(9/33): ceph-mon-17.2.7-397.gb824e5be.el9.x86_6 4.6 MB/s | 3.7 MB 00:00 2024-01-23T08:38:25.846 INFO:teuthology.orchestra.run.smithi059.stdout:(10/33): python3-requests-2.25.1-7.el9.noarch.r 1.6 MB/s | 124 kB 00:00 2024-01-23T08:38:25.964 INFO:teuthology.orchestra.run.smithi059.stdout:(11/33): python3-urllib3-1.26.5-5.el9.noarch.rp 1.5 MB/s | 215 kB 00:00 2024-01-23T08:38:26.122 INFO:teuthology.orchestra.run.smithi059.stdout:(12/33): ceph-osd-17.2.7-397.gb824e5be.el9.x86_ 19 MB/s | 15 MB 00:00 2024-01-23T08:38:26.306 INFO:teuthology.orchestra.run.smithi059.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 102 kB/s | 35 kB 00:00 2024-01-23T08:38:26.431 INFO:teuthology.orchestra.run.smithi059.stdout:(14/33): python3-mako-1.1.4-6.el9.noarch.rpm 295 kB/s | 172 kB 00:00 2024-01-23T08:38:26.464 INFO:teuthology.orchestra.run.smithi059.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 149 kB/s | 51 kB 00:00 2024-01-23T08:38:26.490 INFO:teuthology.orchestra.run.smithi059.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 226 kB/s | 42 kB 00:00 2024-01-23T08:40:35.333 INFO:teuthology.orchestra.run.smithi059.stdout:[MIRROR] python3-bcrypt-3.2.2-1.el9.x86_64.rpm: Curl error (28): Timeout was reached for https://epel.mirror.shastacoe.net/epel/9/Everything/x86_64/Packages/p/python3-bcrypt-3.2.2-1.el9.x86_64.rpm [Failed to connect to epel.mirror.shastacoe.net port 443: Connection timed out] 2024-01-23T08:40:35.334 INFO:teuthology.orchestra.run.smithi059.stdout:[MIRROR] python3-cheroot-8.6.0-4.el9.noarch.rpm: Curl error (28): Timeout was reached for https://epel.mirror.shastacoe.net/epel/9/Everything/x86_64/Packages/p/python3-cheroot-8.6.0-4.el9.noarch.rpm [Failed to connect to epel.mirror.shastacoe.net port 443: Connection timed out] 2024-01-23T08:40:35.334 INFO:teuthology.orchestra.run.smithi059.stdout:[MIRROR] python3-cherrypy-18.6.1-2.el9.noarch.rpm: Curl error (28): Timeout was reached for https://epel.mirror.shastacoe.net/epel/9/Everything/x86_64/Packages/p/python3-cherrypy-18.6.1-2.el9.noarch.rpm [Failed to connect to epel.mirror.shastacoe.net port 443: Connection timed out] 2024-01-23T08:40:35.800 INFO:teuthology.orchestra.run.smithi059.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 344 B/s | 43 kB 02:09 2024-01-23T08:40:35.884 INFO:teuthology.orchestra.run.smithi059.stdout:(18/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 kB/s | 358 kB 02:09 2024-01-23T08:40:35.908 INFO:teuthology.orchestra.run.smithi059.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 kB/s | 172 kB 02:09 2024-01-23T08:40:35.934 INFO:teuthology.orchestra.run.smithi059.stdout:(20/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 80 kB/s | 11 kB 00:00 2024-01-23T08:40:35.967 INFO:teuthology.orchestra.run.smithi059.stdout:(21/33): python3-jaraco-classes-3.2.1-5.el9.noa 212 kB/s | 18 kB 00:00 2024-01-23T08:40:36.026 INFO:teuthology.orchestra.run.smithi059.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 252 kB/s | 23 kB 00:00 2024-01-23T08:40:36.051 INFO:teuthology.orchestra.run.smithi059.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 166 kB/s | 19 kB 00:00 2024-01-23T08:40:36.077 INFO:teuthology.orchestra.run.smithi059.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2024-01-23T08:40:36.110 INFO:teuthology.orchestra.run.smithi059.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 552 kB/s | 46 kB 00:00 2024-01-23T08:40:36.136 INFO:teuthology.orchestra.run.smithi059.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 939 kB/s | 79 kB 00:00 2024-01-23T08:40:36.219 INFO:teuthology.orchestra.run.smithi059.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-01-23T08:40:36.244 INFO:teuthology.orchestra.run.smithi059.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 122 kB/s | 16 kB 00:00 2024-01-23T08:40:36.261 INFO:teuthology.orchestra.run.smithi059.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 717 kB/s | 90 kB 00:00 2024-01-23T08:40:36.303 INFO:teuthology.orchestra.run.smithi059.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 428 kB/s | 36 kB 00:00 2024-01-23T08:40:36.353 INFO:teuthology.orchestra.run.smithi059.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-01-23T08:40:36.412 INFO:teuthology.orchestra.run.smithi059.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.8 MB/s | 427 kB 00:00 2024-01-23T08:40:36.437 INFO:teuthology.orchestra.run.smithi059.stdout:(33/33): python3-zc-lockfile-2.0-10.el9.noarch. 149 kB/s | 20 kB 00:00 2024-01-23T08:40:36.440 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:40:36.441 INFO:teuthology.orchestra.run.smithi059.stdout:Total 208 kB/s | 27 MB 02:12 2024-01-23T08:40:36.545 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-23T08:40:36.584 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-23T08:40:36.585 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-23T08:40:36.960 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-23T08:40:36.961 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-23T08:40:38.215 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-23T08:40:38.274 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-01-23T08:40:38.314 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-01-23T08:40:38.347 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-01-23T08:40:38.447 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-01-23T08:40:38.522 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-01-23T08:40:38.570 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-01-23T08:40:38.605 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-01-23T08:40:38.634 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-01-23T08:40:38.669 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-01-23T08:40:38.702 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-01-23T08:40:38.743 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-01-23T08:40:38.778 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-01-23T08:40:38.812 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-01-23T08:40:38.860 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-01-23T08:40:38.943 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-01-23T08:40:39.092 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-01-23T08:40:39.177 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-01-23T08:40:39.237 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-01-23T08:40:39.275 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 19/33 2024-01-23T08:40:39.334 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-01-23T08:40:39.405 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-01-23T08:40:39.606 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-01-23T08:40:39.671 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-01-23T08:40:39.753 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-01-23T08:40:39.841 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-01-23T08:40:39.887 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-01-23T08:40:40.009 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-01-23T08:40:40.164 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-397.gb824e5be.el9.n 28/33 2024-01-23T08:40:40.188 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-2:17.2.7-397.gb824e5be.el9.x86_64 29/33 2024-01-23T08:40:40.212 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-2:17.2.7-397.gb824e5be.el9.x86_64 29/33 2024-01-23T08:40:40.212 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-23T08:40:40.212 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-23T08:40:40.212 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-23T08:40:40.212 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-23T08:40:40.213 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:40:41.631 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-osd-2:17.2.7-397.gb824e5be.el9.x86_64 30/33 2024-01-23T08:40:41.656 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-osd-2:17.2.7-397.gb824e5be.el9.x86_64 30/33 2024-01-23T08:40:41.656 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-23T08:40:41.657 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-23T08:40:41.657 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-23T08:40:41.657 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-23T08:40:41.657 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:40:42.048 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mon-2:17.2.7-397.gb824e5be.el9.x86_64 31/33 2024-01-23T08:40:42.070 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mon-2:17.2.7-397.gb824e5be.el9.x86_64 31/33 2024-01-23T08:40:42.070 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-23T08:40:42.070 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-23T08:40:42.070 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-23T08:40:42.070 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-23T08:40:42.070 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:40:42.274 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mds-2:17.2.7-397.gb824e5be.el9.x86_64 32/33 2024-01-23T08:40:42.296 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mds-2:17.2.7-397.gb824e5be.el9.x86_64 32/33 2024-01-23T08:40:42.296 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-23T08:40:42.296 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-23T08:40:42.296 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-23T08:40:42.296 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-23T08:40:42.297 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:40:42.360 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-2:17.2.7-397.gb824e5be.el9.x86_64 33/33 2024-01-23T08:40:43.905 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-2:17.2.7-397.gb824e5be.el9.x86_64 33/33 2024-01-23T08:40:43.905 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-2:17.2.7-397.gb824e5be.el9.x86_64 1/33 2024-01-23T08:40:43.905 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mds-2:17.2.7-397.gb824e5be.el9.x86_64 2/33 2024-01-23T08:40:43.905 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-2:17.2.7-397.gb824e5be.el9.x86_64 3/33 2024-01-23T08:40:43.905 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mon-2:17.2.7-397.gb824e5be.el9.x86_64 4/33 2024-01-23T08:40:43.905 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-osd-2:17.2.7-397.gb824e5be.el9.x86_64 5/33 2024-01-23T08:40:43.905 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-397.gb824e5be.el9.n 6/33 2024-01-23T08:40:43.905 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-01-23T08:40:43.905 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-01-23T08:40:43.906 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-01-23T08:40:43.906 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-01-23T08:40:43.906 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 11/33 2024-01-23T08:40:43.906 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-01-23T08:40:43.906 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-01-23T08:40:43.906 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-01-23T08:40:43.906 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-01-23T08:40:43.906 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-01-23T08:40:43.906 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-01-23T08:40:43.906 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-01-23T08:40:43.906 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-01-23T08:40:43.906 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-01-23T08:40:43.906 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-01-23T08:40:43.907 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-01-23T08:40:43.907 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-01-23T08:40:43.907 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-01-23T08:40:43.907 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-01-23T08:40:43.907 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-01-23T08:40:43.907 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-01-23T08:40:43.907 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-01-23T08:40:43.907 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-01-23T08:40:43.907 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-01-23T08:40:43.907 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-01-23T08:40:43.907 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-01-23T08:40:44.997 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-01-23T08:40:44.997 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:40:44.997 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-23T08:40:44.997 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:40:44.997 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:40:44.997 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:40:44.997 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core-2:17.2.7-397.gb824e5be.el9.noarch 2024-01-23T08:40:44.998 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:40:44.998 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:40:44.998 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-23T08:40:44.998 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-23T08:40:44.998 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-23T08:40:44.998 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-23T08:40:44.998 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-23T08:40:44.998 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-23T08:40:44.998 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-23T08:40:44.998 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-23T08:40:44.998 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-23T08:40:44.998 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-23T08:40:44.998 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-23T08:40:44.998 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-23T08:40:44.999 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-23T08:40:44.999 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-23T08:40:44.999 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-23T08:40:44.999 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ply-3.11-14.el9.noarch 2024-01-23T08:40:44.999 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-23T08:40:44.999 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-23T08:40:44.999 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-23T08:40:44.999 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-23T08:40:44.999 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-23T08:40:44.999 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-23T08:40:44.999 INFO:teuthology.orchestra.run.smithi059.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-23T08:40:44.999 INFO:teuthology.orchestra.run.smithi059.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-23T08:40:44.999 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-23T08:40:44.999 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-23T08:40:45.000 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-23T08:40:45.000 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:40:45.000 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:40:45.340 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-base 2024-01-23T08:40:45.945 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:03 ago on Tue 23 Jan 2024 08:37:42 AM UTC. 2024-01-23T08:40:46.252 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:40:46.252 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:40:46.253 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:40:46.253 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.253 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:40:46.253 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.253 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:40:46.253 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.254 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:40:46.254 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.254 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:40:46.254 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.254 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:40:46.254 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.254 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:40:46.254 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.254 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:40:46.254 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.255 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:40:46.255 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.255 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:40:46.255 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.255 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:40:46.255 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.255 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:40:46.255 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.255 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:40:46.255 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.255 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:40:46.255 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.255 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:40:46.255 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.256 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:40:46.256 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.256 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:40:46.256 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.256 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:40:46.256 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.256 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:40:46.256 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.256 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:40:46.256 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.256 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:40:46.256 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.257 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:40:46.257 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.257 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:40:46.257 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.257 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:40:46.257 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.257 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:40:46.257 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.257 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:40:46.257 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.257 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:40:46.257 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.257 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:40:46.258 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.258 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:40:46.258 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.258 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:40:46.258 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.258 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:40:46.258 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.258 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:40:46.258 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.258 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:40:46.258 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.259 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:40:46.259 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.259 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:40:46.259 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.259 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:40:46.259 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.259 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:40:46.259 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.259 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.259 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.259 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:40:46.259 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.259 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.259 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.259 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.260 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:40:46.260 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.260 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.260 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.260 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.260 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.260 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.260 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:40:46.260 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.261 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.261 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.261 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.261 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:40:46.261 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.261 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.261 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.261 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.261 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:40:46.261 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.261 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.262 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.262 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.262 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:40:46.262 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.262 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.262 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.262 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.262 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:40:46.262 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.262 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.262 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.263 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.263 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:40:46.263 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.263 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.263 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.263 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:40:46.263 INFO:teuthology.orchestra.run.smithi059.stdout:Package ceph-base-2:17.2.7-397.gb824e5be.el9.x86_64 is already installed. 2024-01-23T08:40:46.322 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:40:46.323 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-01-23T08:40:46.323 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:40:46.394 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install cephadm 2024-01-23T08:40:46.996 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:04 ago on Tue 23 Jan 2024 08:37:42 AM UTC. 2024-01-23T08:40:47.302 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:40:47.302 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:40:47.302 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:40:47.302 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.302 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:40:47.302 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.303 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:40:47.303 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.303 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:40:47.303 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.303 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:40:47.303 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.303 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:40:47.303 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.303 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:40:47.303 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.303 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:40:47.303 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.303 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:40:47.303 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.304 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:40:47.304 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.304 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:40:47.304 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.304 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:40:47.304 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.304 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:40:47.304 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.304 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:40:47.304 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.304 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:40:47.304 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.304 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:40:47.304 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.305 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:40:47.305 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.305 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:40:47.305 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.305 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:40:47.305 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.305 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:40:47.305 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.305 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:40:47.305 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.305 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:40:47.305 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.305 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:40:47.305 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.305 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:40:47.306 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.306 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:40:47.306 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.306 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:40:47.306 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.306 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:40:47.306 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.306 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:40:47.306 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.306 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:40:47.306 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.306 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:40:47.306 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.307 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:40:47.307 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.307 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:40:47.307 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.307 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:40:47.307 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.307 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:40:47.307 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.307 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:40:47.307 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.307 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:40:47.307 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.307 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:40:47.307 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.308 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.308 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.308 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:40:47.308 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.308 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.308 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.308 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.308 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:40:47.308 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.308 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.308 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.308 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.308 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.308 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.309 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:40:47.309 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.309 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.309 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.309 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.309 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:40:47.309 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.309 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.309 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.309 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.309 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:40:47.309 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.309 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.309 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.310 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.310 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:40:47.310 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.310 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.310 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.310 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.310 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:40:47.310 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.310 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.310 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.310 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.310 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:40:47.310 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.310 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.310 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.311 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:40:47.371 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:40:47.372 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:40:47.372 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-01-23T08:40:47.372 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:40:47.372 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-23T08:40:47.372 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm noarch 2:17.2.7-397.gb824e5be.el9 ceph-noarch 80 k 2024-01-23T08:40:47.373 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:40:47.373 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-23T08:40:47.373 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:40:47.373 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-01-23T08:40:47.373 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:40:47.373 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 80 k 2024-01-23T08:40:47.373 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 364 k 2024-01-23T08:40:47.373 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-23T08:40:47.716 INFO:teuthology.orchestra.run.smithi059.stdout:cephadm-17.2.7-397.gb824e5be.el9.noarch.rpm 235 kB/s | 80 kB 00:00 2024-01-23T08:40:47.716 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:40:47.716 INFO:teuthology.orchestra.run.smithi059.stdout:Total 234 kB/s | 80 kB 00:00 2024-01-23T08:40:47.716 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-23T08:40:47.722 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-23T08:40:47.722 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-23T08:40:47.724 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-23T08:40:47.724 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-23T08:40:47.794 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-23T08:40:48.346 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:17.2.7-397.gb824e5be.el9.noarch 1/1 2024-01-23T08:40:48.422 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephadm-2:17.2.7-397.gb824e5be.el9.noarch 1/1 2024-01-23T08:40:48.795 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:17.2.7-397.gb824e5be.el9.noarch 1/1 2024-01-23T08:40:49.159 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephadm-2:17.2.7-397.gb824e5be.el9.noarch 1/1 2024-01-23T08:40:49.159 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:40:49.159 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-23T08:40:49.159 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm-2:17.2.7-397.gb824e5be.el9.noarch 2024-01-23T08:40:49.159 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:40:49.159 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:40:49.378 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-immutable-object-cache 2024-01-23T08:40:49.988 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:07 ago on Tue 23 Jan 2024 08:37:42 AM UTC. 2024-01-23T08:40:50.292 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:40:50.292 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:40:50.292 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:40:50.292 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.292 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:40:50.292 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.292 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:40:50.292 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.292 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:40:50.293 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.293 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:40:50.293 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.293 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:40:50.293 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.293 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:40:50.293 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.293 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:40:50.293 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.293 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:40:50.293 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.293 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:40:50.293 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.293 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:40:50.294 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.294 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:40:50.294 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.294 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:40:50.294 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.294 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:40:50.294 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.294 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:40:50.294 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.294 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:40:50.294 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.294 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:40:50.294 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.294 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:40:50.295 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.295 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:40:50.295 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.295 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:40:50.295 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.295 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:40:50.295 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.295 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:40:50.295 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.295 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:40:50.295 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.295 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:40:50.295 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.295 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:40:50.296 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.296 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:40:50.296 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.296 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:40:50.296 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.296 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:40:50.296 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.296 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:40:50.296 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.296 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:40:50.296 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.296 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:40:50.297 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.297 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:40:50.297 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.297 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:40:50.297 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.297 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:40:50.297 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.297 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:40:50.297 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.297 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:40:50.297 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.297 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:40:50.297 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.297 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.297 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.298 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:40:50.298 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.298 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.298 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.298 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.298 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:40:50.298 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.298 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.298 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.298 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.298 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.298 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.298 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:40:50.298 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.299 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.299 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.299 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.299 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:40:50.299 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.299 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.299 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.299 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.299 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:40:50.299 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.299 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.299 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.299 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.300 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:40:50.300 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.300 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.300 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.300 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.300 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:40:50.300 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.300 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.300 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.300 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.300 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:40:50.300 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.300 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.300 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.301 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:40:50.363 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:40:50.363 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:40:50.363 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-01-23T08:40:50.364 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:40:50.364 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-23T08:40:50.364 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-397.gb824e5be.el9 ceph 137 k 2024-01-23T08:40:50.364 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:40:50.364 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-23T08:40:50.364 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:40:50.364 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-01-23T08:40:50.364 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:40:50.364 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 137 k 2024-01-23T08:40:50.364 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 379 k 2024-01-23T08:40:50.364 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-23T08:40:50.587 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-immutable-object-cache-17.2.7-397.gb824e5b 615 kB/s | 137 kB 00:00 2024-01-23T08:40:50.588 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:40:50.588 INFO:teuthology.orchestra.run.smithi059.stdout:Total 611 kB/s | 137 kB 00:00 2024-01-23T08:40:50.588 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-23T08:40:50.596 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-23T08:40:50.596 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-23T08:40:50.650 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-23T08:40:50.651 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-23T08:40:50.914 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-23T08:40:50.980 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-397.gb824e5be.e 1/1 2024-01-23T08:40:51.004 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-397.gb824e5be.e 1/1 2024-01-23T08:40:51.004 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-23T08:40:51.004 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-23T08:40:51.004 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:40:52.035 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-397.gb824e5be.e 1/1 2024-01-23T08:40:52.035 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:40:52.035 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-23T08:40:52.035 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:40:52.035 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:40:52.035 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:40:52.282 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr 2024-01-23T08:40:52.902 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:10 ago on Tue 23 Jan 2024 08:37:42 AM UTC. 2024-01-23T08:40:53.212 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:40:53.212 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:40:53.212 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:40:53.212 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.213 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:40:53.213 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.213 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:40:53.213 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.213 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:40:53.213 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.213 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:40:53.213 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.213 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:40:53.213 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.213 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:40:53.213 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.214 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:40:53.214 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.214 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:40:53.214 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.214 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:40:53.214 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.214 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:40:53.214 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.214 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:40:53.214 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.214 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:40:53.214 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.214 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:40:53.214 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.214 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:40:53.215 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.215 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:40:53.215 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.215 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:40:53.215 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.215 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:40:53.215 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.216 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:40:53.216 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.216 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:40:53.216 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.216 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:40:53.216 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.216 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:40:53.216 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.216 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:40:53.216 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.216 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:40:53.216 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.217 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:40:53.217 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.217 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:40:53.217 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.217 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:40:53.217 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.217 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:40:53.217 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.217 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:40:53.217 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.217 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:40:53.217 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.217 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:40:53.217 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.218 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:40:53.218 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.218 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:40:53.218 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.218 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:40:53.218 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.218 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:40:53.218 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.218 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:40:53.218 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.218 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:40:53.218 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.218 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.218 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.218 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:40:53.219 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.219 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.219 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.219 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.219 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:40:53.219 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.219 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.219 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.219 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.219 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.219 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.219 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:40:53.219 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.219 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.219 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.220 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.220 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:40:53.220 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.220 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.220 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.220 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.220 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:40:53.220 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.220 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.220 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.220 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.220 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:40:53.220 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.220 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.220 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.221 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.221 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:40:53.221 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.221 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.221 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.221 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.221 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:40:53.221 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.221 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.221 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.221 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:40:53.222 INFO:teuthology.orchestra.run.smithi059.stdout:Package ceph-mgr-2:17.2.7-397.gb824e5be.el9.x86_64 is already installed. 2024-01-23T08:40:53.283 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:40:53.284 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-01-23T08:40:53.284 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:40:53.354 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-dashboard 2024-01-23T08:40:53.964 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:11 ago on Tue 23 Jan 2024 08:37:42 AM UTC. 2024-01-23T08:40:54.269 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:40:54.269 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:40:54.269 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:40:54.269 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.269 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:40:54.269 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.269 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:40:54.269 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.269 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:40:54.269 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.270 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:40:54.270 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.270 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:40:54.270 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.270 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:40:54.270 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.270 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:40:54.270 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.270 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:40:54.270 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.270 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:40:54.270 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.270 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:40:54.270 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.271 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:40:54.271 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.271 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:40:54.271 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.271 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:40:54.271 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.271 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:40:54.271 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.271 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:40:54.271 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.271 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:40:54.271 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.271 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:40:54.272 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.272 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:40:54.272 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.272 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:40:54.272 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.272 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:40:54.272 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.272 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:40:54.272 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.272 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:40:54.272 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.272 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:40:54.272 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.273 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:40:54.273 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.273 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:40:54.273 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.273 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:40:54.273 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.273 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:40:54.273 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.273 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:40:54.273 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.273 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:40:54.273 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.274 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:40:54.274 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.274 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:40:54.274 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.274 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:40:54.274 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.274 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:40:54.274 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.274 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:40:54.274 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.274 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:40:54.274 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.274 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:40:54.275 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.275 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.275 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.275 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:40:54.275 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.275 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.275 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.275 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.275 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:40:54.275 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.275 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.275 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.275 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.275 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.276 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.276 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:40:54.276 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.276 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.276 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.276 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.276 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:40:54.276 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.276 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.276 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.276 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.276 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:40:54.276 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.276 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.277 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.277 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.277 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:40:54.277 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.277 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.277 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.277 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.277 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:40:54.277 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.277 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.277 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.277 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.277 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:40:54.278 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.278 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.278 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.278 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:40:54.340 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:40:54.341 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:40:54.341 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-01-23T08:40:54.341 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:40:54.341 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-23T08:40:54.341 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard noarch 2:17.2.7-397.gb824e5be.el9 ceph-noarch 1.7 M 2024-01-23T08:40:54.341 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-01-23T08:40:54.341 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards noarch 2:17.2.7-397.gb824e5be.el9 ceph-noarch 23 k 2024-01-23T08:40:54.341 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts noarch 2:17.2.7-397.gb824e5be.el9 ceph-noarch 15 k 2024-01-23T08:40:54.342 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-01-23T08:40:54.342 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-23T08:40:54.342 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-23T08:40:54.342 INFO:teuthology.orchestra.run.smithi059.stdout:Installing weak dependencies: 2024-01-23T08:40:54.342 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-01-23T08:40:54.342 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:40:54.342 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-23T08:40:54.342 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:40:54.342 INFO:teuthology.orchestra.run.smithi059.stdout:Install 7 Packages 2024-01-23T08:40:54.342 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:40:54.343 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 2.0 M 2024-01-23T08:40:54.343 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 8.1 M 2024-01-23T08:40:54.343 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-23T08:40:54.600 INFO:teuthology.orchestra.run.smithi059.stdout:(1/7): ceph-grafana-dashboards-17.2.7-397.gb824 116 kB/s | 23 kB 00:00 2024-01-23T08:40:54.625 INFO:teuthology.orchestra.run.smithi059.stdout:(2/7): ceph-prometheus-alerts-17.2.7-397.gb824e 64 kB/s | 15 kB 00:00 2024-01-23T08:40:54.775 INFO:teuthology.orchestra.run.smithi059.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 52 kB/s | 9.0 kB 00:00 2024-01-23T08:40:54.859 INFO:teuthology.orchestra.run.smithi059.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 175 kB/s | 41 kB 00:00 2024-01-23T08:40:54.884 INFO:teuthology.orchestra.run.smithi059.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 283 kB/s | 31 kB 00:00 2024-01-23T08:40:54.976 INFO:teuthology.orchestra.run.smithi059.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-01-23T08:40:55.118 INFO:teuthology.orchestra.run.smithi059.stdout:(7/7): ceph-mgr-dashboard-17.2.7-397.gb824e5be. 2.4 MB/s | 1.7 MB 00:00 2024-01-23T08:40:55.121 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:40:55.121 INFO:teuthology.orchestra.run.smithi059.stdout:Total 2.6 MB/s | 2.0 MB 00:00 2024-01-23T08:40:55.121 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-23T08:40:55.130 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-23T08:40:55.130 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-23T08:40:55.198 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-23T08:40:55.199 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-23T08:40:55.501 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-23T08:40:55.561 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-01-23T08:40:55.595 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-01-23T08:40:55.642 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-01-23T08:40:55.673 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-01-23T08:40:55.702 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-397.gb824e5be.el9.no 5/7 2024-01-23T08:40:55.942 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-397.gb824e5be.el9.n 6/7 2024-01-23T08:40:55.971 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-397.gb824e5be.el9.noarch 7/7 2024-01-23T08:40:56.646 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-397.gb824e5be.el9.noarch 7/7 2024-01-23T08:40:56.646 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-397.gb824e5be.el9.n 1/7 2024-01-23T08:40:56.646 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-397.gb824e5be.el9.noarch 2/7 2024-01-23T08:40:56.646 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-397.gb824e5be.el9.no 3/7 2024-01-23T08:40:56.646 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-01-23T08:40:56.647 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-01-23T08:40:56.647 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-01-23T08:40:57.092 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-01-23T08:40:57.092 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:40:57.092 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-23T08:40:57.092 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards-2:17.2.7-397.gb824e5be.el9.noarch 2024-01-23T08:40:57.092 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard-2:17.2.7-397.gb824e5be.el9.noarch 2024-01-23T08:40:57.092 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts-2:17.2.7-397.gb824e5be.el9.noarch 2024-01-23T08:40:57.092 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-01-23T08:40:57.092 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-01-23T08:40:57.092 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-23T08:40:57.092 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-23T08:40:57.093 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:40:57.093 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:40:57.327 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-23T08:40:57.938 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:15 ago on Tue 23 Jan 2024 08:37:42 AM UTC. 2024-01-23T08:40:58.253 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:40:58.253 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:40:58.253 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:40:58.253 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.253 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:40:58.254 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.254 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:40:58.254 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.254 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:40:58.254 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.254 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:40:58.254 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.254 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:40:58.254 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.254 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:40:58.254 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.254 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:40:58.255 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.255 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:40:58.255 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.255 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:40:58.255 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.255 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:40:58.255 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.255 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:40:58.255 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.255 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:40:58.255 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.255 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:40:58.255 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.255 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:40:58.256 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.256 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:40:58.256 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.256 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:40:58.256 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.256 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:40:58.256 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.256 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:40:58.256 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.256 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:40:58.256 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.256 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:40:58.256 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.257 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:40:58.257 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.257 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:40:58.257 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.257 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:40:58.257 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.257 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:40:58.257 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.257 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:40:58.257 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.257 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:40:58.257 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.257 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:40:58.257 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.258 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:40:58.258 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.258 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:40:58.258 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.258 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:40:58.258 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.258 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:40:58.258 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.258 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:40:58.258 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.258 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:40:58.258 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.258 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:40:58.258 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.258 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:40:58.259 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.259 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:40:58.259 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.259 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.259 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.259 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:40:58.259 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.259 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.259 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.259 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.259 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:40:58.259 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.259 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.260 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.260 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.260 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.260 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.260 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:40:58.260 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.260 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.260 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.260 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.260 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:40:58.260 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.260 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.260 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.260 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.261 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:40:58.261 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.261 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.261 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.261 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.261 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:40:58.261 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.261 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.261 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.261 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.261 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:40:58.261 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.261 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.261 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.262 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.262 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:40:58.262 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.262 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.262 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.262 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:40:58.326 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:40:58.327 INFO:teuthology.orchestra.run.smithi059.stdout:====================================================================================== 2024-01-23T08:40:58.327 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-01-23T08:40:58.327 INFO:teuthology.orchestra.run.smithi059.stdout:====================================================================================== 2024-01-23T08:40:58.328 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-23T08:40:58.328 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-397.gb824e5be.el9 ceph-noarch 7.4 M 2024-01-23T08:40:58.328 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-01-23T08:40:58.328 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-01-23T08:40:58.328 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-01-23T08:40:58.328 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-01-23T08:40:58.328 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-01-23T08:40:58.328 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-01-23T08:40:58.328 INFO:teuthology.orchestra.run.smithi059.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-01-23T08:40:58.328 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-01-23T08:40:58.328 INFO:teuthology.orchestra.run.smithi059.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-23T08:40:58.328 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-01-23T08:40:58.329 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-01-23T08:40:58.329 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-01-23T08:40:58.329 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:40:58.329 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-23T08:40:58.329 INFO:teuthology.orchestra.run.smithi059.stdout:====================================================================================== 2024-01-23T08:40:58.329 INFO:teuthology.orchestra.run.smithi059.stdout:Install 12 Packages 2024-01-23T08:40:58.329 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:40:58.329 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 38 M 2024-01-23T08:40:58.330 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 209 M 2024-01-23T08:40:58.330 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-23T08:40:58.933 INFO:teuthology.orchestra.run.smithi059.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-01-23T08:40:58.983 INFO:teuthology.orchestra.run.smithi059.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 656 kB/s | 33 kB 00:00 2024-01-23T08:40:59.008 INFO:teuthology.orchestra.run.smithi059.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.5 MB/s | 803 kB 00:00 2024-01-23T08:40:59.050 INFO:teuthology.orchestra.run.smithi059.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 439 kB/s | 18 kB 00:00 2024-01-23T08:40:59.092 INFO:teuthology.orchestra.run.smithi059.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 845 kB/s | 35 kB 00:00 2024-01-23T08:40:59.142 INFO:teuthology.orchestra.run.smithi059.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 19 MB/s | 3.0 MB 00:00 2024-01-23T08:40:59.193 INFO:teuthology.orchestra.run.smithi059.stdout:(7/12): python3-devel-3.9.17-2.el9.x86_64.rpm 4.8 MB/s | 245 kB 00:00 2024-01-23T08:40:59.260 INFO:teuthology.orchestra.run.smithi059.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2024-01-23T08:40:59.310 INFO:teuthology.orchestra.run.smithi059.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.9 MB/s | 250 kB 00:00 2024-01-23T08:40:59.377 INFO:teuthology.orchestra.run.smithi059.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 28 MB/s | 5.1 MB 00:00 2024-01-23T08:40:59.710 INFO:teuthology.orchestra.run.smithi059.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-01-23T08:41:00.252 INFO:teuthology.orchestra.run.smithi059.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-3 5.0 MB/s | 7.4 MB 00:01 2024-01-23T08:41:00.253 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:41:00.253 INFO:teuthology.orchestra.run.smithi059.stdout:Total 20 MB/s | 38 MB 00:01 2024-01-23T08:41:00.612 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-23T08:41:00.639 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-23T08:41:00.639 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-23T08:41:01.148 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-23T08:41:01.148 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-23T08:41:01.941 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-23T08:41:02.023 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-01-23T08:41:02.071 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-01-23T08:41:02.174 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-01-23T08:41:02.209 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-01-23T08:41:02.394 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-23T08:41:02.426 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-23T08:41:02.509 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-01-23T08:41:03.196 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-01-23T08:41:03.309 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-01-23T08:41:04.626 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-01-23T08:41:05.704 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-01-23T08:41:05.729 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-397.gb824e5 12/12 2024-01-23T08:41:08.237 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-397.gb824e5 12/12 2024-01-23T08:41:08.237 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-397.gb824e5 1/12 2024-01-23T08:41:08.238 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-01-23T08:41:08.238 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-01-23T08:41:08.238 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-01-23T08:41:08.238 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-01-23T08:41:08.238 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-01-23T08:41:08.238 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-01-23T08:41:08.238 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-01-23T08:41:08.238 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 9/12 2024-01-23T08:41:08.238 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-01-23T08:41:08.238 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-01-23T08:41:08.696 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-01-23T08:41:08.696 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:08.696 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-23T08:41:08.696 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local-2:17.2.7-397.gb824e5be.el9.noarch 2024-01-23T08:41:08.696 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-23T08:41:08.696 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-23T08:41:08.696 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-23T08:41:08.696 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-23T08:41:08.696 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-23T08:41:08.696 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-23T08:41:08.697 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-23T08:41:08.697 INFO:teuthology.orchestra.run.smithi059.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-23T08:41:08.697 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-23T08:41:08.697 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-23T08:41:08.697 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-23T08:41:08.697 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:08.697 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:41:09.081 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-rook 2024-01-23T08:41:09.692 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:27 ago on Tue 23 Jan 2024 08:37:42 AM UTC. 2024-01-23T08:41:09.996 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:41:09.996 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:41:09.996 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:41:09.996 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:41:09.996 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:41:09.996 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:41:09.997 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:41:09.997 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:41:09.997 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:41:09.997 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:09.997 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:41:09.997 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:41:09.997 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:41:09.997 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:09.997 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:41:09.997 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:41:09.997 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:41:09.997 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:41:09.997 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:41:09.997 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:41:09.998 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:41:09.998 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:09.998 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:41:09.998 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:41:09.998 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:41:09.998 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:41:09.998 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:41:09.998 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:09.998 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:41:09.998 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:41:09.998 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:41:09.998 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:09.998 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:41:09.998 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:09.999 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:41:09.999 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:09.999 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:41:09.999 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:09.999 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:41:09.999 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:09.999 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:41:09.999 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:09.999 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:41:09.999 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:41:09.999 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:41:09.999 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:09.999 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:41:09.999 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.000 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:41:10.000 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.000 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:41:10.000 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.000 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:41:10.000 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.000 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:41:10.000 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.000 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:41:10.000 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.000 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:41:10.000 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.000 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:41:10.000 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.000 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:41:10.001 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.001 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:41:10.001 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.001 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:41:10.001 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.001 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:41:10.001 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.001 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:41:10.001 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.001 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:41:10.001 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.001 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:41:10.001 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.001 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.002 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.002 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:41:10.002 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.002 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.002 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.002 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.002 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:41:10.002 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.002 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.002 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.002 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.002 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.002 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.002 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:41:10.002 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.003 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.003 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.003 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.003 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:41:10.003 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.003 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.003 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.003 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.003 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:41:10.003 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.003 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.003 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.003 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.003 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:41:10.004 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.004 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.004 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.004 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.004 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:41:10.004 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.004 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.004 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.004 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.004 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:41:10.004 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.004 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.004 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.005 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:41:10.067 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:41:10.068 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:41:10.069 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-01-23T08:41:10.069 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:41:10.069 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-23T08:41:10.069 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook noarch 2:17.2.7-397.gb824e5be.el9 ceph-noarch 47 k 2024-01-23T08:41:10.069 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-01-23T08:41:10.069 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-23T08:41:10.069 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-23T08:41:10.069 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth noarch 1:2.26.2-1.el9 epel 202 k 2024-01-23T08:41:10.069 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-01-23T08:41:10.069 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-01-23T08:41:10.069 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-23T08:41:10.069 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-01-23T08:41:10.069 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-01-23T08:41:10.070 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-01-23T08:41:10.070 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-01-23T08:41:10.070 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-23T08:41:10.070 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-23T08:41:10.070 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:10.070 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-23T08:41:10.070 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:41:10.070 INFO:teuthology.orchestra.run.smithi059.stdout:Install 13 Packages 2024-01-23T08:41:10.070 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:10.071 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 2.2 M 2024-01-23T08:41:10.071 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 26 M 2024-01-23T08:41:10.071 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-23T08:41:10.436 INFO:teuthology.orchestra.run.smithi059.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 183 kB/s | 26 kB 00:00 2024-01-23T08:41:10.461 INFO:teuthology.orchestra.run.smithi059.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 116 kB/s | 19 kB 00:00 2024-01-23T08:41:10.595 INFO:teuthology.orchestra.run.smithi059.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 222 kB 00:00 2024-01-23T08:41:10.620 INFO:teuthology.orchestra.run.smithi059.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-01-23T08:41:10.671 INFO:teuthology.orchestra.run.smithi059.stdout:(5/13): ceph-mgr-rook-17.2.7-397.gb824e5be.el9. 127 kB/s | 47 kB 00:00 2024-01-23T08:41:10.763 INFO:teuthology.orchestra.run.smithi059.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 463 kB/s | 54 kB 00:00 2024-01-23T08:41:10.788 INFO:teuthology.orchestra.run.smithi059.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.4 MB/s | 279 kB 00:00 2024-01-23T08:41:11.488 INFO:teuthology.orchestra.run.smithi059.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 21 kB/s | 15 kB 00:00 2024-01-23T08:41:11.638 INFO:teuthology.orchestra.run.smithi059.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 34 kB/s | 32 kB 00:00 2024-01-23T08:41:12.297 INFO:teuthology.orchestra.run.smithi059.stdout:(10/13): python3-google-auth-2.26.2-1.el9.noarc 134 kB/s | 202 kB 00:01 2024-01-23T08:41:12.322 INFO:teuthology.orchestra.run.smithi059.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 86 kB/s | 59 kB 00:00 2024-01-23T08:41:12.684 INFO:teuthology.orchestra.run.smithi059.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 877 kB/s | 1.0 MB 00:01 2024-01-23T08:41:12.705 INFO:teuthology.orchestra.run.smithi059.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 220 kB/s | 90 kB 00:00 2024-01-23T08:41:12.709 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:41:12.710 INFO:teuthology.orchestra.run.smithi059.stdout:Total 852 kB/s | 2.2 MB 00:02 2024-01-23T08:41:12.779 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-23T08:41:12.795 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-23T08:41:12.795 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-23T08:41:12.905 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-23T08:41:12.905 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-23T08:41:13.440 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-23T08:41:13.628 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-01-23T08:41:13.684 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-01-23T08:41:13.740 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-01-23T08:41:13.775 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-01-23T08:41:13.817 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-01-23T08:41:13.915 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-01-23T08:41:14.025 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-google-auth-1:2.26.2-1.el9.noarch 7/13 2024-01-23T08:41:14.077 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-01-23T08:41:14.855 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-01-23T08:41:14.906 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-01-23T08:41:14.940 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-01-23T08:41:15.003 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-01-23T08:41:15.034 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-rook-2:17.2.7-397.gb824e5be.el9.noarch 13/13 2024-01-23T08:41:15.862 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-397.gb824e5be.el9.noarch 13/13 2024-01-23T08:41:15.862 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-rook-2:17.2.7-397.gb824e5be.el9.noarch 1/13 2024-01-23T08:41:15.862 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-01-23T08:41:15.862 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-01-23T08:41:15.862 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-01-23T08:41:15.862 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-01-23T08:41:15.862 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-01-23T08:41:15.863 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-01-23T08:41:15.863 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-01-23T08:41:15.863 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-01-23T08:41:15.863 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-google-auth-1:2.26.2-1.el9.noarch 10/13 2024-01-23T08:41:15.863 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-01-23T08:41:15.863 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-01-23T08:41:16.257 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-01-23T08:41:16.257 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:16.257 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-23T08:41:16.258 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook-2:17.2.7-397.gb824e5be.el9.noarch 2024-01-23T08:41:16.258 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-23T08:41:16.258 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-23T08:41:16.258 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth-1:2.26.2-1.el9.noarch 2024-01-23T08:41:16.258 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-23T08:41:16.258 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-23T08:41:16.258 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-23T08:41:16.258 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-23T08:41:16.258 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-23T08:41:16.258 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-23T08:41:16.258 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-23T08:41:16.258 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-23T08:41:16.258 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-23T08:41:16.258 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:16.259 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:41:16.527 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-cephadm 2024-01-23T08:41:17.137 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:35 ago on Tue 23 Jan 2024 08:37:42 AM UTC. 2024-01-23T08:41:17.447 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:41:17.447 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:41:17.448 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:41:17.448 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.448 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:41:17.448 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.448 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:41:17.448 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.448 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:41:17.448 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.448 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:41:17.448 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.448 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:41:17.448 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.448 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:41:17.449 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.449 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:41:17.449 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.449 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:41:17.449 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.449 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:41:17.449 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.449 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:41:17.449 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.449 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:41:17.449 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.449 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:41:17.449 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.450 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:41:17.450 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.450 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:41:17.450 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.450 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:41:17.450 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.450 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:41:17.450 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.450 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:41:17.450 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.450 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:41:17.450 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.450 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:41:17.450 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.451 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:41:17.451 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.451 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:41:17.451 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.451 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:41:17.451 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.451 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:41:17.451 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.451 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:41:17.451 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.451 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:41:17.451 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.451 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:41:17.451 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.452 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:41:17.452 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.452 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:41:17.452 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.452 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:41:17.452 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.452 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:41:17.452 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.452 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:41:17.452 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.452 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:41:17.452 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.452 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:41:17.453 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.453 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:41:17.453 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.453 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:41:17.453 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.453 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:41:17.453 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.453 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.453 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.453 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:41:17.453 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.453 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.453 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.453 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.454 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:41:17.454 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.454 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.454 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.454 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.454 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.454 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.454 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:41:17.454 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.454 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.454 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.454 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.454 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:41:17.454 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.455 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.455 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.455 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.455 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:41:17.455 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.455 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.455 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.455 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.455 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:41:17.455 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.455 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.455 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.455 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.455 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:41:17.456 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.456 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.456 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.456 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.456 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:41:17.456 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.456 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.456 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.456 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:41:17.519 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:41:17.520 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:41:17.520 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-01-23T08:41:17.520 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:41:17.520 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-23T08:41:17.520 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm noarch 2:17.2.7-397.gb824e5be.el9 ceph-noarch 121 k 2024-01-23T08:41:17.520 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-01-23T08:41:17.521 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-23T08:41:17.521 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-01-23T08:41:17.521 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2 noarch 2.11.3-4.el9 appstream 249 k 2024-01-23T08:41:17.521 INFO:teuthology.orchestra.run.smithi059.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-23T08:41:17.521 INFO:teuthology.orchestra.run.smithi059.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-23T08:41:17.521 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:17.521 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-23T08:41:17.521 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:41:17.521 INFO:teuthology.orchestra.run.smithi059.stdout:Install 6 Packages 2024-01-23T08:41:17.521 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:17.522 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 7.0 M 2024-01-23T08:41:17.522 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 33 M 2024-01-23T08:41:17.522 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-23T08:41:18.148 INFO:teuthology.orchestra.run.smithi059.stdout:(1/6): ceph-mgr-cephadm-17.2.7-397.gb824e5be.el 342 kB/s | 121 kB 00:00 2024-01-23T08:41:18.298 INFO:teuthology.orchestra.run.smithi059.stdout:(2/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 494 kB/s | 249 kB 00:00 2024-01-23T08:41:18.323 INFO:teuthology.orchestra.run.smithi059.stdout:(3/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 3.1 MB/s | 548 kB 00:00 2024-01-23T08:41:18.365 INFO:teuthology.orchestra.run.smithi059.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.2 MB/s | 51 kB 00:00 2024-01-23T08:41:18.399 INFO:teuthology.orchestra.run.smithi059.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 574 kB/s | 58 kB 00:00 2024-01-23T08:41:20.724 INFO:teuthology.orchestra.run.smithi059.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.0 MB/s | 6.0 MB 00:02 2024-01-23T08:41:20.724 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:41:20.724 INFO:teuthology.orchestra.run.smithi059.stdout:Total 2.2 MB/s | 7.0 MB 00:03 2024-01-23T08:41:20.798 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-23T08:41:20.810 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-23T08:41:20.811 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-23T08:41:20.920 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-23T08:41:20.922 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-23T08:41:21.246 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-23T08:41:21.428 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-01-23T08:41:21.476 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-01-23T08:41:22.029 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-01-23T08:41:22.123 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-01-23T08:41:22.184 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-01-23T08:41:22.221 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-397.gb824e5be.el9.noarch 6/6 2024-01-23T08:41:22.957 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-397.gb824e5be.el9.noarch 6/6 2024-01-23T08:41:22.957 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-397.gb824e5be.el9.noarch 1/6 2024-01-23T08:41:22.957 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-01-23T08:41:22.957 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-01-23T08:41:22.957 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-01-23T08:41:22.957 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-01-23T08:41:23.288 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-01-23T08:41:23.288 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:23.289 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-23T08:41:23.289 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm-2:17.2.7-397.gb824e5be.el9.noarch 2024-01-23T08:41:23.289 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-23T08:41:23.289 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-23T08:41:23.289 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-23T08:41:23.289 INFO:teuthology.orchestra.run.smithi059.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-23T08:41:23.289 INFO:teuthology.orchestra.run.smithi059.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-23T08:41:23.289 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:23.289 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:41:23.553 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-fuse 2024-01-23T08:41:24.159 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:42 ago on Tue 23 Jan 2024 08:37:42 AM UTC. 2024-01-23T08:41:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:41:24.469 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:41:24.470 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:41:24.470 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.470 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:41:24.470 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.470 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:41:24.470 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.470 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:41:24.470 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.470 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:41:24.470 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.470 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:41:24.470 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.470 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:41:24.471 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.471 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:41:24.471 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.471 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:41:24.471 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.471 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:41:24.471 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.471 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:41:24.471 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.471 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:41:24.471 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.471 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:41:24.471 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.471 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:41:24.472 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.472 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:41:24.472 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.472 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:41:24.472 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.472 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:41:24.472 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.472 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:41:24.472 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.472 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:41:24.472 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.472 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:41:24.472 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.473 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:41:24.473 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.473 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:41:24.473 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.473 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:41:24.473 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.473 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:41:24.473 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.473 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:41:24.473 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.473 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:41:24.473 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.473 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:41:24.473 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.474 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:41:24.474 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.474 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:41:24.474 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.474 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:41:24.474 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.474 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:41:24.474 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.474 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:41:24.474 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.474 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:41:24.474 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.474 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:41:24.474 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.474 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:41:24.474 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.475 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:41:24.475 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.475 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:41:24.475 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.475 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.475 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.475 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:41:24.475 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.475 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.475 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.475 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.475 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:41:24.475 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.475 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.476 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.476 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.476 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.476 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.476 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:41:24.476 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.476 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.476 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.476 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.476 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:41:24.476 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.476 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.476 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.476 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.477 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:41:24.477 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.477 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.477 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.477 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.477 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:41:24.477 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.477 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.477 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.477 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.477 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:41:24.477 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.477 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.477 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.478 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.478 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:41:24.478 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.478 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.478 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.478 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:41:24.541 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:41:24.542 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:41:24.542 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-01-23T08:41:24.542 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:41:24.542 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-23T08:41:24.542 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse x86_64 2:17.2.7-397.gb824e5be.el9 ceph 810 k 2024-01-23T08:41:24.542 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-01-23T08:41:24.542 INFO:teuthology.orchestra.run.smithi059.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-01-23T08:41:24.542 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:24.542 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-23T08:41:24.542 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:41:24.542 INFO:teuthology.orchestra.run.smithi059.stdout:Install 2 Packages 2024-01-23T08:41:24.542 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:24.543 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 891 k 2024-01-23T08:41:24.543 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 2.6 M 2024-01-23T08:41:24.543 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-23T08:41:25.100 INFO:teuthology.orchestra.run.smithi059.stdout:(1/2): ceph-fuse-17.2.7-397.gb824e5be.el9.x86_6 1.9 MB/s | 810 kB 00:00 2024-01-23T08:41:25.125 INFO:teuthology.orchestra.run.smithi059.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 177 kB/s | 80 kB 00:00 2024-01-23T08:41:25.125 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:41:25.126 INFO:teuthology.orchestra.run.smithi059.stdout:Total 1.5 MB/s | 891 kB 00:00 2024-01-23T08:41:25.137 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-23T08:41:25.161 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-23T08:41:25.161 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-23T08:41:25.243 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-23T08:41:25.243 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-23T08:41:25.431 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-23T08:41:25.557 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-01-23T08:41:25.627 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-fuse-2:17.2.7-397.gb824e5be.el9.x86_64 2/2 2024-01-23T08:41:26.339 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-fuse-2:17.2.7-397.gb824e5be.el9.x86_64 2/2 2024-01-23T08:41:26.340 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-fuse-2:17.2.7-397.gb824e5be.el9.x86_64 1/2 2024-01-23T08:41:26.704 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-01-23T08:41:26.704 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:26.704 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-23T08:41:26.704 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse-2:17.2.7-397.gb824e5be.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-01-23T08:41:26.705 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:26.705 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:41:26.979 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librados-devel 2024-01-23T08:41:27.599 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:45 ago on Tue 23 Jan 2024 08:37:42 AM UTC. 2024-01-23T08:41:27.902 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:41:27.902 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:41:27.903 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:41:27.903 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.903 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:41:27.903 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.903 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:41:27.903 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.903 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:41:27.903 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.903 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:41:27.903 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.903 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:41:27.903 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.903 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:41:27.904 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.904 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:41:27.904 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.904 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:41:27.904 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.904 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:41:27.904 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.904 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:41:27.904 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.904 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:41:27.904 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.904 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:41:27.905 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.905 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:41:27.905 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.905 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:41:27.905 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.905 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:41:27.905 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.905 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:41:27.905 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.905 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:41:27.905 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.905 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:41:27.906 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.906 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:41:27.906 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.906 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:41:27.906 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.906 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:41:27.906 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.906 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:41:27.906 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.906 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:41:27.906 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.906 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:41:27.906 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.907 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:41:27.907 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.907 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:41:27.907 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.907 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:41:27.907 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.907 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:41:27.907 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.907 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:41:27.907 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.908 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:41:27.908 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.908 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:41:27.908 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.908 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:41:27.908 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.908 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:41:27.908 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.908 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:41:27.908 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.908 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:41:27.909 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.909 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:41:27.909 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.909 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.909 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.909 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:41:27.909 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.909 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.909 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.909 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.909 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:41:27.909 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.909 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.910 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.910 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.910 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.910 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.910 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:41:27.910 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.910 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.910 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.910 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.910 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:41:27.910 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.910 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.911 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.911 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.911 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:41:27.911 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.911 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.911 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.911 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.911 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:41:27.911 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.911 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.911 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.911 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.912 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:41:27.912 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.912 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.912 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.912 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.912 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:41:27.912 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.912 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.912 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.912 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:41:27.973 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:41:27.974 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:41:27.974 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-01-23T08:41:27.974 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:41:27.974 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-23T08:41:27.974 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel x86_64 2:17.2.7-397.gb824e5be.el9 ceph 126 k 2024-01-23T08:41:27.974 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:27.974 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-23T08:41:27.974 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:41:27.974 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-01-23T08:41:27.974 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:27.974 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 126 k 2024-01-23T08:41:27.975 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 448 k 2024-01-23T08:41:27.975 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-23T08:41:28.295 INFO:teuthology.orchestra.run.smithi059.stdout:librados-devel-17.2.7-397.gb824e5be.el9.x86_64. 394 kB/s | 126 kB 00:00 2024-01-23T08:41:28.296 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:41:28.296 INFO:teuthology.orchestra.run.smithi059.stdout:Total 392 kB/s | 126 kB 00:00 2024-01-23T08:41:28.296 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-23T08:41:28.302 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-23T08:41:28.302 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-23T08:41:28.364 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-23T08:41:28.364 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-23T08:41:28.518 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-23T08:41:28.629 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librados-devel-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:41:29.100 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados-devel-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:41:29.454 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados-devel-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:41:29.454 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:29.454 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-23T08:41:29.454 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:41:29.455 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:29.455 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:41:29.704 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install libcephfs2 2024-01-23T08:41:30.314 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:48 ago on Tue 23 Jan 2024 08:37:42 AM UTC. 2024-01-23T08:41:30.620 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:41:30.620 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:41:30.620 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:41:30.620 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.620 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:41:30.621 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.621 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:41:30.621 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.621 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:41:30.621 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.621 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:41:30.621 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.621 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:41:30.621 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.621 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:41:30.621 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.621 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:41:30.621 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.621 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:41:30.622 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.622 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:41:30.622 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.622 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:41:30.622 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.622 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:41:30.622 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.622 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:41:30.622 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.622 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:41:30.622 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.622 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:41:30.622 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.622 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:41:30.623 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.623 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:41:30.623 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.623 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:41:30.623 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.623 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:41:30.623 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.623 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:41:30.623 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.623 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:41:30.623 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.623 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:41:30.623 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.623 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:41:30.624 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.624 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:41:30.624 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.624 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:41:30.624 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.624 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:41:30.624 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.624 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:41:30.624 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.624 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:41:30.624 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.624 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:41:30.625 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.625 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:41:30.625 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.625 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:41:30.625 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.625 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:41:30.625 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.625 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:41:30.625 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.625 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:41:30.625 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.625 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:41:30.625 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.625 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:41:30.625 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.626 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:41:30.626 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.626 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.626 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.626 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:41:30.626 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.626 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.626 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.626 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.626 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:41:30.626 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.626 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.626 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.626 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.627 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.627 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.627 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:41:30.627 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.627 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.627 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.627 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.627 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:41:30.627 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.627 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.627 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.627 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.627 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:41:30.627 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.628 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.628 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.628 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.628 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:41:30.628 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.628 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.628 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.628 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.628 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:41:30.628 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.628 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.628 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.628 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.628 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:41:30.629 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.629 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.629 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.629 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:41:30.629 INFO:teuthology.orchestra.run.smithi059.stdout:Package libcephfs2-2:17.2.7-397.gb824e5be.el9.x86_64 is already installed. 2024-01-23T08:41:30.691 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:41:30.692 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-01-23T08:41:30.692 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:41:30.765 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install libcephfs-devel 2024-01-23T08:41:31.378 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:49 ago on Tue 23 Jan 2024 08:37:42 AM UTC. 2024-01-23T08:41:31.684 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:41:31.684 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:41:31.684 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:41:31.684 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.684 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:41:31.684 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.684 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:41:31.684 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.684 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:41:31.684 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.685 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:41:31.685 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.685 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:41:31.685 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.685 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:41:31.685 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.685 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:41:31.685 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.685 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:41:31.685 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.685 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:41:31.685 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.685 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:41:31.685 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.686 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:41:31.686 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.686 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:41:31.686 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.686 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:41:31.686 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.686 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:41:31.686 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.686 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:41:31.686 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.686 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:41:31.686 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.686 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:41:31.686 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.686 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:41:31.687 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.687 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:41:31.687 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.687 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:41:31.687 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.687 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:41:31.687 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.687 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:41:31.687 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.687 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:41:31.687 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.687 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:41:31.687 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.687 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:41:31.688 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.688 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:41:31.688 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.688 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:41:31.688 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.688 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:41:31.688 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.688 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:41:31.688 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.688 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:41:31.688 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.688 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:41:31.688 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.689 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:41:31.689 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.689 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:41:31.689 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.689 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:41:31.689 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.689 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:41:31.689 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.689 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:41:31.689 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.689 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.689 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.689 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:41:31.689 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.690 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.690 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.690 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.690 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:41:31.690 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.690 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.690 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.690 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.690 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.690 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.690 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:41:31.690 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.690 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.690 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.691 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.691 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:41:31.691 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.691 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.691 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.691 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.691 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:41:31.691 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.691 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.691 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.691 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.691 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:41:31.691 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.691 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.691 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.692 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.692 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:41:31.692 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.692 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.692 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.692 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.692 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:41:31.692 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.692 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.692 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.692 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:41:31.757 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:41:31.757 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:41:31.758 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-01-23T08:41:31.758 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:41:31.758 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-23T08:41:31.758 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel x86_64 2:17.2.7-397.gb824e5be.el9 ceph 25 k 2024-01-23T08:41:31.758 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:31.758 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-23T08:41:31.758 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:41:31.758 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-01-23T08:41:31.758 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:31.758 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 25 k 2024-01-23T08:41:31.758 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 110 k 2024-01-23T08:41:31.758 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-23T08:41:31.998 INFO:teuthology.orchestra.run.smithi059.stdout:libcephfs-devel-17.2.7-397.gb824e5be.el9.x86_64 104 kB/s | 25 kB 00:00 2024-01-23T08:41:31.999 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:41:31.999 INFO:teuthology.orchestra.run.smithi059.stdout:Total 103 kB/s | 25 kB 00:00 2024-01-23T08:41:31.999 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-23T08:41:32.003 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-23T08:41:32.003 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-23T08:41:32.004 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-23T08:41:32.005 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-23T08:41:32.092 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-23T08:41:32.192 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs-devel-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:41:32.553 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:41:32.855 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs-devel-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:41:32.855 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:32.855 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-23T08:41:32.855 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:41:32.855 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:32.856 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:41:33.049 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librados2 2024-01-23T08:41:33.657 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:51 ago on Tue 23 Jan 2024 08:37:42 AM UTC. 2024-01-23T08:41:33.974 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:41:33.974 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:41:33.974 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:41:33.974 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.974 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:41:33.974 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.975 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:41:33.975 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.975 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:41:33.975 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.975 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:41:33.975 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.975 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:41:33.975 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.975 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:41:33.975 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.975 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:41:33.975 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.975 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:41:33.975 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.976 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:41:33.976 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.976 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:41:33.976 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.976 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:41:33.976 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.976 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:41:33.976 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.976 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:41:33.976 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.976 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:41:33.976 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.976 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:41:33.976 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.977 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:41:33.977 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.977 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:41:33.977 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.977 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:41:33.977 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.977 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:41:33.977 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.977 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:41:33.977 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.977 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:41:33.977 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.977 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:41:33.977 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.978 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:41:33.978 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.978 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:41:33.978 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.978 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:41:33.978 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.978 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:41:33.978 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.978 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:41:33.978 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.978 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:41:33.978 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.978 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:41:33.978 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.978 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:41:33.979 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.979 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:41:33.979 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.979 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:41:33.979 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.979 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:41:33.979 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.979 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:41:33.979 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.979 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:41:33.980 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.980 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:41:33.980 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.980 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.980 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.980 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:41:33.980 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.980 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.980 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.980 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.980 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:41:33.980 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.980 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.980 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.981 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.981 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.981 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.981 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:41:33.981 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.981 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.981 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.981 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.981 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:41:33.981 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.981 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.981 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.981 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.982 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:41:33.982 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.982 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.982 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.982 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.982 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:41:33.982 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.982 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.982 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.982 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.982 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:41:33.982 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.983 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.983 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.983 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.983 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:41:33.983 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.983 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.983 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.983 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:41:33.984 INFO:teuthology.orchestra.run.smithi059.stdout:Package librados2-2:17.2.7-397.gb824e5be.el9.x86_64 is already installed. 2024-01-23T08:41:34.045 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:41:34.047 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-01-23T08:41:34.047 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:41:34.123 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librbd1 2024-01-23T08:41:34.732 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:52 ago on Tue 23 Jan 2024 08:37:42 AM UTC. 2024-01-23T08:41:35.054 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:41:35.054 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:41:35.054 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:41:35.055 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.055 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:41:35.055 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.055 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:41:35.055 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.055 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:41:35.055 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.055 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:41:35.055 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.055 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:41:35.055 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.055 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:41:35.055 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.055 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:41:35.056 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.056 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:41:35.056 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.056 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:41:35.056 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.056 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:41:35.056 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.056 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:41:35.056 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.056 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:41:35.056 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.056 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:41:35.056 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.056 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:41:35.057 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.057 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:41:35.057 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.057 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:41:35.057 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.057 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:41:35.057 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.057 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:41:35.057 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.057 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:41:35.057 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.057 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:41:35.057 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.057 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:41:35.058 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.058 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:41:35.058 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.058 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:41:35.058 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.058 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:41:35.058 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.058 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:41:35.058 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.058 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:41:35.058 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.058 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:41:35.058 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.058 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:41:35.059 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.059 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:41:35.059 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.059 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:41:35.059 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.059 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:41:35.059 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.059 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:41:35.059 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.059 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:41:35.059 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.059 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:41:35.059 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.059 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:41:35.059 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.060 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:41:35.060 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.060 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.060 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.060 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:41:35.060 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.060 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.060 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.060 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.060 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:41:35.060 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.060 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.060 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.060 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.061 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.061 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.061 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:41:35.061 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.061 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.061 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.061 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.061 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:41:35.061 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.061 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.061 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.061 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.061 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:41:35.061 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.061 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.062 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.062 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.062 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:41:35.062 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.062 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.062 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.062 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.062 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:41:35.062 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.062 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.062 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.062 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.062 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:41:35.062 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.062 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.063 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.063 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:41:35.063 INFO:teuthology.orchestra.run.smithi059.stdout:Package librbd1-2:17.2.7-397.gb824e5be.el9.x86_64 is already installed. 2024-01-23T08:41:35.127 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:41:35.128 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-01-23T08:41:35.128 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:41:35.210 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rados 2024-01-23T08:41:35.816 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:53 ago on Tue 23 Jan 2024 08:37:42 AM UTC. 2024-01-23T08:41:36.125 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:41:36.125 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:41:36.126 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:41:36.126 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.126 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:41:36.126 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.126 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:41:36.126 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.126 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:41:36.126 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.126 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:41:36.126 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.126 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:41:36.127 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.127 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:41:36.127 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.127 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:41:36.127 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.127 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:41:36.127 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.127 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:41:36.127 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.127 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:41:36.128 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.128 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:41:36.128 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.128 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:41:36.128 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.128 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:41:36.128 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.128 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:41:36.128 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.128 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:41:36.128 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.128 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:41:36.129 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.129 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:41:36.129 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.129 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:41:36.129 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.129 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:41:36.129 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.129 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:41:36.129 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.129 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:41:36.129 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.129 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:41:36.129 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.129 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:41:36.130 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.130 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:41:36.130 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.130 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:41:36.130 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.130 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:41:36.130 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.130 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:41:36.130 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.130 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:41:36.130 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.130 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:41:36.130 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.130 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:41:36.131 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.131 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:41:36.131 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.131 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:41:36.131 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.131 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:41:36.131 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.131 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:41:36.131 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.131 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:41:36.131 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.131 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:41:36.131 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.131 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.132 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.132 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:41:36.132 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.132 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.132 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.132 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.132 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:41:36.132 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.132 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.132 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.132 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.132 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.133 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.133 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:41:36.133 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.133 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.133 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.133 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.134 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:41:36.134 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.134 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.134 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.134 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.134 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:41:36.134 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.134 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.134 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.135 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.135 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:41:36.135 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.135 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.135 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.135 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.135 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:41:36.135 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.135 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.135 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.135 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.135 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:41:36.135 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.135 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.136 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.136 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:41:36.136 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rados-2:17.2.7-397.gb824e5be.el9.x86_64 is already installed. 2024-01-23T08:41:36.197 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:41:36.198 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-01-23T08:41:36.198 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:41:36.273 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rgw 2024-01-23T08:41:36.882 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:54 ago on Tue 23 Jan 2024 08:37:42 AM UTC. 2024-01-23T08:41:37.188 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:41:37.188 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:41:37.188 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:41:37.188 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.188 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:41:37.189 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.189 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:41:37.189 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.189 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:41:37.189 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.189 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:41:37.189 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.189 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:41:37.189 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.189 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:41:37.189 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.189 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:41:37.189 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.189 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:41:37.190 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.190 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:41:37.190 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.190 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:41:37.190 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.190 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:41:37.190 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.190 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:41:37.190 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.190 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:41:37.190 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.190 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:41:37.190 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.190 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:41:37.191 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.191 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:41:37.191 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.191 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:41:37.191 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.191 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:41:37.191 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.191 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:41:37.191 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.191 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:41:37.191 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.191 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:41:37.191 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.192 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:41:37.192 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.192 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:41:37.192 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.192 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:41:37.192 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.192 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:41:37.192 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.192 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:41:37.192 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.192 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:41:37.192 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.192 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:41:37.192 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.193 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:41:37.193 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.193 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:41:37.193 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.193 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:41:37.193 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.193 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:41:37.193 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.193 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:41:37.193 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.193 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:41:37.193 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.193 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:41:37.193 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.194 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:41:37.194 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.194 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.194 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.194 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:41:37.194 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.194 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.194 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.194 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.194 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:41:37.194 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.194 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.194 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.195 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.195 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.195 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.195 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:41:37.195 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.195 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.195 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.195 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.195 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:41:37.195 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.195 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.195 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.195 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.195 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:41:37.196 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.196 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.196 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.196 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.196 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:41:37.196 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.196 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.196 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.196 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.196 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:41:37.196 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.196 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.196 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.197 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.197 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:41:37.197 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.197 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.197 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.197 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:41:37.197 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rgw-2:17.2.7-397.gb824e5be.el9.x86_64 is already installed. 2024-01-23T08:41:37.259 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:41:37.260 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-01-23T08:41:37.260 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:41:37.351 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-cephfs 2024-01-23T08:41:37.967 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:55 ago on Tue 23 Jan 2024 08:37:42 AM UTC. 2024-01-23T08:41:38.272 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:41:38.273 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:41:38.273 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:41:38.273 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.273 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:41:38.273 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.273 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:41:38.273 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.273 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:41:38.273 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.273 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:41:38.273 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.273 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:41:38.273 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.274 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:41:38.274 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.274 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:41:38.274 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.274 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:41:38.274 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.274 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:41:38.274 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.274 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:41:38.274 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.274 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:41:38.274 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.275 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:41:38.275 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.275 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:41:38.275 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.275 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:41:38.275 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.275 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:41:38.275 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.275 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:41:38.275 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.275 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:41:38.275 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.275 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:41:38.275 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.276 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:41:38.276 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.276 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:41:38.276 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.276 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:41:38.276 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.276 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:41:38.276 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.276 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:41:38.276 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.276 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:41:38.276 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.276 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:41:38.277 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.277 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:41:38.277 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.277 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:41:38.277 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.277 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:41:38.277 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.277 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:41:38.277 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.277 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:41:38.277 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.277 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:41:38.277 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.277 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:41:38.277 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.277 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:41:38.278 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.278 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:41:38.278 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.278 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:41:38.278 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.278 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:41:38.278 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.278 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.278 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.278 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:41:38.278 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.279 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.279 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.279 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.279 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:41:38.279 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.279 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.279 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.279 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.279 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.279 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.279 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:41:38.279 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.279 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.279 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.279 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.280 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:41:38.280 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.280 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.280 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.280 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.280 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:41:38.280 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.280 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.280 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.280 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.280 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:41:38.280 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.280 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.280 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.281 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.281 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:41:38.281 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.281 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.281 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.281 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.281 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:41:38.281 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.281 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.281 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.281 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:41:38.282 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-cephfs-2:17.2.7-397.gb824e5be.el9.x86_64 is already installed. 2024-01-23T08:41:38.346 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:41:38.347 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-01-23T08:41:38.347 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:41:38.425 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rbd 2024-01-23T08:41:39.026 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:57 ago on Tue 23 Jan 2024 08:37:42 AM UTC. 2024-01-23T08:41:39.332 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:41:39.332 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:41:39.332 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:41:39.333 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.333 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:41:39.333 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.333 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:41:39.333 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.333 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:41:39.333 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.333 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:41:39.333 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.333 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:41:39.333 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.333 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:41:39.333 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.334 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:41:39.334 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.334 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:41:39.334 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.334 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:41:39.334 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.334 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:41:39.334 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.334 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:41:39.334 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.334 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:41:39.334 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.334 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:41:39.335 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.335 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:41:39.335 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.335 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:41:39.335 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.335 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:41:39.335 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.335 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:41:39.335 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.335 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:41:39.335 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.335 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:41:39.335 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.335 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:41:39.336 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.336 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:41:39.336 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.336 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:41:39.336 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.336 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:41:39.336 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.336 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:41:39.336 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.336 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:41:39.336 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.336 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:41:39.336 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.336 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:41:39.337 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.337 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:41:39.337 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.337 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:41:39.337 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.337 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:41:39.337 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.337 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:41:39.337 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.337 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:41:39.337 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.337 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:41:39.337 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.337 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:41:39.338 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.338 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:41:39.338 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.338 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:41:39.338 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.338 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.338 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.338 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:41:39.338 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.338 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.338 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.338 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.338 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:41:39.338 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.339 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.339 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.339 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.339 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.339 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.339 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:41:39.339 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.339 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.339 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.339 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.339 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:41:39.339 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.339 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.339 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.340 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.340 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:41:39.340 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.340 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.340 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.340 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.340 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:41:39.340 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.340 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.340 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.340 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.340 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:41:39.340 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.340 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.341 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.341 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.341 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:41:39.341 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.341 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.341 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.341 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:41:39.341 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rbd-2:17.2.7-397.gb824e5be.el9.x86_64 is already installed. 2024-01-23T08:41:39.404 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:41:39.405 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-01-23T08:41:39.405 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:41:39.479 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-fuse 2024-01-23T08:41:40.083 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:58 ago on Tue 23 Jan 2024 08:37:42 AM UTC. 2024-01-23T08:41:40.390 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:41:40.390 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:41:40.390 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:41:40.390 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.390 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:41:40.390 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.390 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:41:40.391 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.391 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:41:40.391 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.391 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:41:40.391 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.391 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:41:40.391 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.391 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:41:40.391 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.391 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:41:40.391 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.391 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:41:40.391 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.392 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:41:40.392 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.392 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:41:40.392 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.392 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:41:40.392 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.392 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:41:40.392 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.392 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:41:40.392 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.392 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:41:40.392 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.393 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:41:40.393 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.393 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:41:40.393 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.393 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:41:40.393 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.393 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:41:40.393 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.393 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:41:40.393 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.394 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:41:40.394 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.394 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:41:40.394 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.394 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:41:40.394 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.394 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:41:40.394 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.394 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:41:40.394 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.395 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:41:40.395 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.395 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:41:40.395 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.395 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:41:40.395 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.395 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:41:40.395 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.395 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:41:40.395 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.395 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:41:40.395 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.395 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:41:40.396 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.396 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:41:40.396 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.396 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:41:40.396 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.396 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:41:40.396 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.396 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:41:40.396 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.396 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:41:40.396 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.396 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.397 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.397 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:41:40.397 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.397 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.397 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.397 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.397 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:41:40.397 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.397 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.397 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.397 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.397 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.398 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.398 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:41:40.398 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.398 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.398 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.398 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.399 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:41:40.399 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.399 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.399 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.399 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.399 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:41:40.399 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.399 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.399 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.399 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.399 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:41:40.399 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.399 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.399 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.399 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.399 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:41:40.400 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.400 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.400 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.400 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.400 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:41:40.400 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.400 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.400 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.400 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:41:40.464 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:41:40.466 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:41:40.466 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-01-23T08:41:40.466 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:41:40.466 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-23T08:41:40.466 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse x86_64 2:17.2.7-397.gb824e5be.el9 ceph 81 k 2024-01-23T08:41:40.466 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:40.466 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-23T08:41:40.466 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:41:40.466 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-01-23T08:41:40.466 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:40.466 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 81 k 2024-01-23T08:41:40.466 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 222 k 2024-01-23T08:41:40.466 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-23T08:41:40.685 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-fuse-17.2.7-397.gb824e5be.el9.x86_64.rpm 373 kB/s | 81 kB 00:00 2024-01-23T08:41:40.685 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:41:40.686 INFO:teuthology.orchestra.run.smithi059.stdout:Total 371 kB/s | 81 kB 00:00 2024-01-23T08:41:40.686 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-23T08:41:40.692 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-23T08:41:40.693 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-23T08:41:40.753 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-23T08:41:40.754 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-23T08:41:40.897 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-23T08:41:41.010 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-fuse-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:41:41.402 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-fuse-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:41:41.737 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-fuse-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:41:41.737 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:41.737 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-23T08:41:41.737 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:41:41.737 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:41.737 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:41:41.985 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-mirror 2024-01-23T08:41:42.595 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:04:00 ago on Tue 23 Jan 2024 08:37:42 AM UTC. 2024-01-23T08:41:42.908 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:41:42.909 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:41:42.909 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:41:42.909 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.909 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:41:42.909 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.909 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:41:42.909 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.909 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:41:42.909 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.910 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:41:42.910 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.910 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:41:42.910 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.910 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:41:42.910 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.910 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:41:42.910 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.910 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:41:42.910 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.910 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:41:42.910 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.910 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:41:42.910 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.910 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:41:42.911 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.911 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:41:42.911 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.911 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:41:42.911 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.911 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:41:42.911 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.911 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:41:42.911 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.911 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:41:42.911 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.911 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:41:42.911 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.912 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:41:42.912 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.912 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:41:42.912 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.912 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:41:42.912 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.912 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:41:42.912 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.912 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:41:42.912 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.912 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:41:42.912 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.912 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:41:42.913 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.913 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:41:42.913 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.913 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:41:42.913 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.913 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:41:42.913 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.913 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:41:42.913 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.913 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:41:42.913 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.913 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:41:42.913 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.914 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:41:42.914 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.914 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:41:42.914 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.914 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:41:42.914 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.914 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:41:42.914 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.914 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:41:42.914 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.914 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:41:42.914 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.914 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.914 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.915 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:41:42.915 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.915 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.915 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.915 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.915 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:41:42.915 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.915 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.915 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.915 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.915 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.915 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.915 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:41:42.915 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.916 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.916 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.916 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.916 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:41:42.916 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.916 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.916 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.916 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.916 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:41:42.916 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.916 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.916 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.916 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.917 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:41:42.917 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.917 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.917 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.917 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.917 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:41:42.917 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.917 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.917 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.917 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.917 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:41:42.917 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.917 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.917 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.918 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:41:42.980 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:41:42.981 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:41:42.981 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-01-23T08:41:42.981 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:41:42.981 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-23T08:41:42.981 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror x86_64 2:17.2.7-397.gb824e5be.el9 ceph 2.9 M 2024-01-23T08:41:42.981 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:42.981 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-23T08:41:42.981 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:41:42.981 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-01-23T08:41:42.981 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:42.982 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 2.9 M 2024-01-23T08:41:42.982 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 12 M 2024-01-23T08:41:42.982 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-23T08:41:43.420 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-mirror-17.2.7-397.gb824e5be.el9.x86_64.rpm 6.7 MB/s | 2.9 MB 00:00 2024-01-23T08:41:43.421 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:41:43.421 INFO:teuthology.orchestra.run.smithi059.stdout:Total 6.7 MB/s | 2.9 MB 00:00 2024-01-23T08:41:43.421 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-23T08:41:43.431 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-23T08:41:43.431 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-23T08:41:43.510 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-23T08:41:43.510 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-23T08:41:43.937 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-23T08:41:43.988 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-mirror-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:41:44.011 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-mirror-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:41:44.011 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-23T08:41:44.011 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-23T08:41:44.011 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-23T08:41:44.011 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-23T08:41:44.011 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:45.144 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-mirror-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:41:45.144 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:45.144 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-23T08:41:45.144 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:41:45.144 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:45.144 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:41:45.423 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-nbd 2024-01-23T08:41:46.032 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:04:04 ago on Tue 23 Jan 2024 08:37:42 AM UTC. 2024-01-23T08:41:46.346 INFO:teuthology.orchestra.run.smithi059.stderr:Modular dependency problems: 2024-01-23T08:41:46.346 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T08:41:46.346 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 1: conflicting requests 2024-01-23T08:41:46.346 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.346 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 2: conflicting requests 2024-01-23T08:41:46.346 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.346 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 3: conflicting requests 2024-01-23T08:41:46.346 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.346 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 4: conflicting requests 2024-01-23T08:41:46.347 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.347 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 5: conflicting requests 2024-01-23T08:41:46.347 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.347 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 6: conflicting requests 2024-01-23T08:41:46.347 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.347 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 7: conflicting requests 2024-01-23T08:41:46.347 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.347 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 8: conflicting requests 2024-01-23T08:41:46.347 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.347 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 9: conflicting requests 2024-01-23T08:41:46.347 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.347 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 10: conflicting requests 2024-01-23T08:41:46.347 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.348 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 11: conflicting requests 2024-01-23T08:41:46.348 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.348 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 12: conflicting requests 2024-01-23T08:41:46.348 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.348 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 13: conflicting requests 2024-01-23T08:41:46.348 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.348 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 14: conflicting requests 2024-01-23T08:41:46.348 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.348 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 15: conflicting requests 2024-01-23T08:41:46.348 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.348 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 16: conflicting requests 2024-01-23T08:41:46.348 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.348 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 17: conflicting requests 2024-01-23T08:41:46.349 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.349 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 18: conflicting requests 2024-01-23T08:41:46.349 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.349 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 19: conflicting requests 2024-01-23T08:41:46.349 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.349 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 20: conflicting requests 2024-01-23T08:41:46.349 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.349 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 21: conflicting requests 2024-01-23T08:41:46.349 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.349 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 22: conflicting requests 2024-01-23T08:41:46.349 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.349 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 23: conflicting requests 2024-01-23T08:41:46.349 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.350 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 24: conflicting requests 2024-01-23T08:41:46.350 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.350 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 25: conflicting requests 2024-01-23T08:41:46.350 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.350 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 26: conflicting requests 2024-01-23T08:41:46.350 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240117165712:17377f89.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.350 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 27: conflicting requests 2024-01-23T08:41:46.350 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.350 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 28: conflicting requests 2024-01-23T08:41:46.350 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.350 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 29: conflicting requests 2024-01-23T08:41:46.350 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.350 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 30: conflicting requests 2024-01-23T08:41:46.350 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.351 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 31: conflicting requests 2024-01-23T08:41:46.351 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.351 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 32: conflicting requests 2024-01-23T08:41:46.351 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.351 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 33: conflicting requests 2024-01-23T08:41:46.351 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.351 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 34: conflicting requests 2024-01-23T08:41:46.351 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.351 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 35: conflicting requests 2024-01-23T08:41:46.351 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.351 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 36: conflicting requests 2024-01-23T08:41:46.351 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240117192000:9edba152.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.351 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 37: conflicting requests 2024-01-23T08:41:46.351 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.351 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.352 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.352 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 38: cannot install the best candidate for the job 2024-01-23T08:41:46.352 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.352 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.352 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.352 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.352 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 39: conflicting requests 2024-01-23T08:41:46.352 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.352 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.352 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.352 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.352 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.352 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.352 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 40: cannot install the best candidate for the job 2024-01-23T08:41:46.353 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.353 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.353 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.353 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.353 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 41: cannot install the best candidate for the job 2024-01-23T08:41:46.353 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.353 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.353 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.353 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.353 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 42: cannot install the best candidate for the job 2024-01-23T08:41:46.354 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.354 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.354 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.354 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.354 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 43: cannot install the best candidate for the job 2024-01-23T08:41:46.354 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.354 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.354 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.354 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.354 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 44: cannot install the best candidate for the job 2024-01-23T08:41:46.354 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.354 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.354 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.355 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.355 INFO:teuthology.orchestra.run.smithi059.stderr: Problem 45: cannot install the best candidate for the job 2024-01-23T08:41:46.355 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.355 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.355 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.355 INFO:teuthology.orchestra.run.smithi059.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-23T08:41:46.417 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-01-23T08:41:46.418 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:41:46.418 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-01-23T08:41:46.418 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:41:46.418 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-01-23T08:41:46.418 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd x86_64 2:17.2.7-397.gb824e5be.el9 ceph 164 k 2024-01-23T08:41:46.418 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:46.418 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-01-23T08:41:46.418 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-01-23T08:41:46.418 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-01-23T08:41:46.418 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:46.418 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 164 k 2024-01-23T08:41:46.418 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 477 k 2024-01-23T08:41:46.418 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-01-23T08:41:46.773 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-nbd-17.2.7-397.gb824e5be.el9.x86_64.rpm 463 kB/s | 164 kB 00:00 2024-01-23T08:41:46.774 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:41:46.774 INFO:teuthology.orchestra.run.smithi059.stdout:Total 461 kB/s | 164 kB 00:00 2024-01-23T08:41:46.774 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-01-23T08:41:46.783 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-01-23T08:41:46.783 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-01-23T08:41:46.845 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-01-23T08:41:46.845 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-01-23T08:41:46.994 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-01-23T08:41:47.092 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-nbd-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:41:47.517 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-nbd-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:41:47.854 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-nbd-2:17.2.7-397.gb824e5be.el9.x86_64 1/1 2024-01-23T08:41:47.854 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:47.854 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-01-23T08:41:47.855 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd-2:17.2.7-397.gb824e5be.el9.x86_64 2024-01-23T08:41:47.855 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:41:47.855 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-01-23T08:41:48.120 DEBUG:teuthology.parallel:result is None 2024-01-23T08:41:48.121 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-23T08:41:48.121 INFO:teuthology.packaging:ref: None 2024-01-23T08:41:48.121 INFO:teuthology.packaging:tag: None 2024-01-23T08:41:48.121 INFO:teuthology.packaging:branch: quincy 2024-01-23T08:41:48.121 INFO:teuthology.packaging:sha1: 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-23T08:41:48.121 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-01-23T08:41:48.381 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-23T08:41:48.401 INFO:teuthology.orchestra.run.smithi059.stdout:17.2.7-397.gb824e5be.el9 2024-01-23T08:41:48.402 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-397.gb824e5be.el9 2024-01-23T08:41:48.402 INFO:teuthology.task.install:The correct ceph version 17.2.7-397.gb824e5be is installed. 2024-01-23T08:41:48.404 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-23T08:41:48.404 INFO:teuthology.packaging:ref: None 2024-01-23T08:41:48.404 INFO:teuthology.packaging:tag: None 2024-01-23T08:41:48.404 INFO:teuthology.packaging:branch: quincy 2024-01-23T08:41:48.404 INFO:teuthology.packaging:sha1: 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-23T08:41:48.404 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-01-23T08:41:48.595 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-23T08:41:48.621 INFO:teuthology.orchestra.run.smithi137.stdout:17.2.7-397.gb824e5be.el9 2024-01-23T08:41:48.621 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-397.gb824e5be.el9 2024-01-23T08:41:48.621 INFO:teuthology.task.install:The correct ceph version 17.2.7-397.gb824e5be is installed. 2024-01-23T08:41:48.623 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-23T08:41:48.623 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-23T08:41:48.623 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-23T08:41:48.651 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-23T08:41:48.651 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-23T08:41:48.694 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-23T08:41:48.694 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-23T08:41:48.694 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/daemon-helper 2024-01-23T08:41:48.722 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-23T08:41:48.788 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-23T08:41:48.789 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/daemon-helper 2024-01-23T08:41:48.819 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-23T08:41:48.888 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-23T08:41:48.889 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-23T08:41:48.889 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-23T08:41:48.915 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-23T08:41:48.981 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-23T08:41:48.981 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-23T08:41:49.011 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-23T08:41:49.079 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-23T08:41:49.079 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-23T08:41:49.079 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/stdin-killer 2024-01-23T08:41:49.106 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-23T08:41:49.172 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-23T08:41:49.172 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/stdin-killer 2024-01-23T08:41:49.199 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-23T08:41:49.267 INFO:teuthology.run_tasks:Running task print... 2024-01-23T08:41:49.279 INFO:teuthology.task.print:**** done install task... 2024-01-23T08:41:49.279 INFO:teuthology.run_tasks:Running task cephadm... 2024-01-23T08:41:49.413 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d'} 2024-01-23T08:41:49.413 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-01-23T08:41:49.414 INFO:tasks.cephadm:Cluster fsid is 3fc07a38-b9cb-11ee-95b1-87774f69a715 2024-01-23T08:41:49.414 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-01-23T08:41:49.414 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-01-23T08:41:49.414 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi059': '172.21.15.59', 'mon.smithi137': '172.21.15.137'} 2024-01-23T08:41:49.415 INFO:tasks.cephadm:Normalizing hostnames... 2024-01-23T08:41:49.415 DEBUG:teuthology.orchestra.run.smithi059:> sudo hostname $(hostname -s) 2024-01-23T08:41:49.443 DEBUG:teuthology.orchestra.run.smithi137:> sudo hostname $(hostname -s) 2024-01-23T08:41:49.471 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-01-23T08:41:49.471 DEBUG:teuthology.orchestra.run.smithi059:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-23T08:41:49.808 INFO:teuthology.orchestra.run.smithi059.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Jan 23 08:41 /home/ubuntu/cephtest/cephadm 2024-01-23T08:41:49.808 DEBUG:teuthology.orchestra.run.smithi137:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-23T08:41:50.086 INFO:teuthology.orchestra.run.smithi137.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Jan 23 08:41 /home/ubuntu/cephtest/cephadm 2024-01-23T08:41:50.086 DEBUG:teuthology.orchestra.run.smithi059:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-23T08:41:50.102 DEBUG:teuthology.orchestra.run.smithi137:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-23T08:41:50.128 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-01-23T08:41:50.129 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-01-23T08:41:50.145 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-01-23T08:41:51.008 INFO:teuthology.orchestra.run.smithi059.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-01-23T08:41:51.075 INFO:teuthology.orchestra.run.smithi137.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-01-23T08:42:23.060 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-01-23T08:42:23.061 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_version": "ceph version 17.2.7-397-gb824e5be (b824e5befce1d427fda2bc90ae52665f0c7e2770) quincy (stable)", 2024-01-23T08:42:23.061 INFO:teuthology.orchestra.run.smithi059.stdout: "image_id": "61b360e6ff650187c031550cf8bb1c0d3e28fc466cffb29cb06fa0da2544d680", 2024-01-23T08:42:23.061 INFO:teuthology.orchestra.run.smithi059.stdout: "repo_digests": [ 2024-01-23T08:42:23.061 INFO:teuthology.orchestra.run.smithi059.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001" 2024-01-23T08:42:23.061 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-01-23T08:42:23.061 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-01-23T08:42:47.220 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-01-23T08:42:47.221 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 17.2.7-397-gb824e5be (b824e5befce1d427fda2bc90ae52665f0c7e2770) quincy (stable)", 2024-01-23T08:42:47.221 INFO:teuthology.orchestra.run.smithi137.stdout: "image_id": "61b360e6ff650187c031550cf8bb1c0d3e28fc466cffb29cb06fa0da2544d680", 2024-01-23T08:42:47.221 INFO:teuthology.orchestra.run.smithi137.stdout: "repo_digests": [ 2024-01-23T08:42:47.221 INFO:teuthology.orchestra.run.smithi137.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001" 2024-01-23T08:42:47.221 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-01-23T08:42:47.221 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-01-23T08:42:47.239 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph 2024-01-23T08:42:47.275 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /etc/ceph 2024-01-23T08:42:47.304 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 777 /etc/ceph 2024-01-23T08:42:47.346 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 777 /etc/ceph 2024-01-23T08:42:47.372 INFO:tasks.cephadm:Writing seed config... 2024-01-23T08:42:47.373 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-01-23T08:42:47.373 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-01-23T08:42:47.374 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-01-23T08:42:47.374 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-01-23T08:42:47.374 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-01-23T08:42:47.374 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-01-23T08:42:47.374 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-01-23T08:42:47.374 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-01-23T08:42:47.374 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-01-23T08:42:47.374 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-01-23T08:42:47.374 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-01-23T08:42:47.374 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-01-23T08:42:47.374 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-01-23T08:42:47.374 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-01-23T08:42:47.374 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-01-23T08:42:47.375 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-01-23T08:42:47.375 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-01-23T08:42:47.375 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-01-23T08:42:47.375 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-01-23T08:42:47.375 INFO:tasks.cephadm: override: [client] debug client = 20 2024-01-23T08:42:47.375 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-01-23T08:42:47.375 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-01-23T08:42:47.375 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-01-23T08:42:47.375 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-01-23T08:42:47.375 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-01-23T08:42:47.375 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-01-23T08:42:47.375 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-01-23T08:42:47.376 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-01-23T08:42:47.376 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-01-23T08:42:47.376 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-01-23T08:42:47.376 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-01-23T08:42:47.376 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-01-23T08:42:47.376 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-01-23T08:42:47.376 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-01-23T08:42:47.376 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-01-23T08:42:47.376 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-01-23T08:42:47.376 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-01-23T08:42:47.376 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-01-23T08:42:47.376 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-01-23T08:42:47.376 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-01-23T08:42:47.377 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-23T08:42:47.377 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-01-23T08:42:47.404 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 3fc07a38-b9cb-11ee-95b1-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-01-23T08:42:47.404 DEBUG:teuthology.orchestra.run.smithi059:mon.smithi059> sudo journalctl -f -n 0 -u ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@mon.smithi059.service 2024-01-23T08:42:47.447 INFO:tasks.cephadm:Bootstrapping... 2024-01-23T08:42:47.447 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.59 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-01-23T08:42:47.659 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-01-23T08:42:47.659 INFO:teuthology.orchestra.run.smithi059.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '3fc07a38-b9cb-11ee-95b1-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.59', '--skip-admin-label'] 2024-01-23T08:42:47.686 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stdout 4.8.1 2024-01-23T08:42:47.686 INFO:teuthology.orchestra.run.smithi059.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-01-23T08:42:47.687 INFO:teuthology.orchestra.run.smithi059.stdout:Verifying podman|docker is present... 2024-01-23T08:42:47.710 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stdout 4.8.1 2024-01-23T08:42:47.711 INFO:teuthology.orchestra.run.smithi059.stdout:Verifying lvm2 is present... 2024-01-23T08:42:47.711 INFO:teuthology.orchestra.run.smithi059.stdout:Verifying time synchronization is in place... 2024-01-23T08:42:47.719 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-01-23T08:42:47.719 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-01-23T08:42:47.727 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-01-23T08:42:47.727 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout inactive 2024-01-23T08:42:47.734 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout enabled 2024-01-23T08:42:47.742 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout active 2024-01-23T08:42:47.742 INFO:teuthology.orchestra.run.smithi059.stdout:Unit chronyd.service is enabled and running 2024-01-23T08:42:47.742 INFO:teuthology.orchestra.run.smithi059.stdout:Repeating the final host check... 2024-01-23T08:42:47.771 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stdout 4.8.1 2024-01-23T08:42:47.772 INFO:teuthology.orchestra.run.smithi059.stdout:podman (/bin/podman) version 4.8.1 is present 2024-01-23T08:42:47.772 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl is present 2024-01-23T08:42:47.772 INFO:teuthology.orchestra.run.smithi059.stdout:lvcreate is present 2024-01-23T08:42:47.779 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-01-23T08:42:47.779 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-01-23T08:42:47.786 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-01-23T08:42:47.786 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout inactive 2024-01-23T08:42:47.794 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout enabled 2024-01-23T08:42:47.802 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout active 2024-01-23T08:42:47.802 INFO:teuthology.orchestra.run.smithi059.stdout:Unit chronyd.service is enabled and running 2024-01-23T08:42:47.802 INFO:teuthology.orchestra.run.smithi059.stdout:Host looks OK 2024-01-23T08:42:47.802 INFO:teuthology.orchestra.run.smithi059.stdout:Cluster fsid: 3fc07a38-b9cb-11ee-95b1-87774f69a715 2024-01-23T08:42:47.802 INFO:teuthology.orchestra.run.smithi059.stdout:Acquiring lock 140571568472128 on /run/cephadm/3fc07a38-b9cb-11ee-95b1-87774f69a715.lock 2024-01-23T08:42:47.802 INFO:teuthology.orchestra.run.smithi059.stdout:Lock 140571568472128 acquired on /run/cephadm/3fc07a38-b9cb-11ee-95b1-87774f69a715.lock 2024-01-23T08:42:47.802 INFO:teuthology.orchestra.run.smithi059.stdout:Verifying IP 172.21.15.59 port 3300 ... 2024-01-23T08:42:47.803 INFO:teuthology.orchestra.run.smithi059.stdout:Verifying IP 172.21.15.59 port 6789 ... 2024-01-23T08:42:47.803 INFO:teuthology.orchestra.run.smithi059.stdout:Base mon IP(s) is [172.21.15.59:3300, 172.21.15.59:6789], mon addrv is [v2:172.21.15.59:3300,v1:172.21.15.59:6789] 2024-01-23T08:42:47.808 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.59 metric 100 2024-01-23T08:42:47.809 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.59 metric 100 2024-01-23T08:42:47.812 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-01-23T08:42:47.812 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-01-23T08:42:47.812 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-01-23T08:42:47.816 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-01-23T08:42:47.816 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-01-23T08:42:47.816 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-01-23T08:42:47.816 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-01-23T08:42:47.816 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-01-23T08:42:47.816 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-01-23T08:42:47.817 INFO:teuthology.orchestra.run.smithi059.stdout:Mon IP `172.21.15.59` is in CIDR network `172.21.0.0/20` 2024-01-23T08:42:47.817 INFO:teuthology.orchestra.run.smithi059.stdout:Mon IP `172.21.15.59` is in CIDR network `172.21.0.0/20` 2024-01-23T08:42:47.817 INFO:teuthology.orchestra.run.smithi059.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-01-23T08:42:47.818 INFO:teuthology.orchestra.run.smithi059.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-01-23T08:42:47.820 INFO:teuthology.orchestra.run.smithi059.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-01-23T08:42:50.860 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stdout 61b360e6ff650187c031550cf8bb1c0d3e28fc466cffb29cb06fa0da2544d680 2024-01-23T08:42:50.860 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-01-23T08:42:50.861 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stderr Getting image source signatures 2024-01-23T08:42:50.861 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stderr Copying blob sha256:b991899f77c549743783eebd6b9410d91b8b606b7c4a49c7af80bbccfdfc40b3 2024-01-23T08:42:50.861 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stderr Copying blob sha256:5892adb860b19d9bc0e53e97111c29aa2896ac907826d7e196bc8387f7e5b09b 2024-01-23T08:42:50.861 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stderr Copying config sha256:61b360e6ff650187c031550cf8bb1c0d3e28fc466cffb29cb06fa0da2544d680 2024-01-23T08:42:50.861 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stderr Writing manifest to image destination 2024-01-23T08:42:57.675 INFO:teuthology.orchestra.run.smithi059.stdout:ceph: stdout ceph version 17.2.7-397-gb824e5be (b824e5befce1d427fda2bc90ae52665f0c7e2770) quincy (stable) 2024-01-23T08:42:57.675 INFO:teuthology.orchestra.run.smithi059.stdout:Ceph version: ceph version 17.2.7-397-gb824e5be (b824e5befce1d427fda2bc90ae52665f0c7e2770) quincy (stable) 2024-01-23T08:42:57.675 INFO:teuthology.orchestra.run.smithi059.stdout:Extracting ceph user uid/gid from container image... 2024-01-23T08:42:58.538 INFO:teuthology.orchestra.run.smithi059.stdout:stat: stdout 167 167 2024-01-23T08:42:58.538 INFO:teuthology.orchestra.run.smithi059.stdout:Creating initial keys... 2024-01-23T08:42:59.443 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph-authtool: stdout AQASfK9lhnd7MhAAx4GSlqyLoy4EDLhHZ8MIWw== 2024-01-23T08:43:00.308 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph-authtool: stdout AQATfK9lnXdgLBAAQolQfbuWAkXWh0/aAShOFg== 2024-01-23T08:43:01.196 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph-authtool: stdout AQAUfK9lcDpIJhAA8Pq5zpKJv6qqS0ekB1Hjcg== 2024-01-23T08:43:01.196 INFO:teuthology.orchestra.run.smithi059.stdout:Creating initial monmap... 2024-01-23T08:43:02.069 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-23T08:43:02.069 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-01-23T08:43:02.069 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 3fc07a38-b9cb-11ee-95b1-87774f69a715 2024-01-23T08:43:02.069 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-23T08:43:02.069 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool for smithi059 [v2:172.21.15.59:3300,v1:172.21.15.59:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-23T08:43:02.069 INFO:teuthology.orchestra.run.smithi059.stdout:setting min_mon_release = octopus 2024-01-23T08:43:02.070 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/monmaptool: set fsid to 3fc07a38-b9cb-11ee-95b1-87774f69a715 2024-01-23T08:43:02.070 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-23T08:43:02.070 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:43:02.070 INFO:teuthology.orchestra.run.smithi059.stdout:Creating mon... 2024-01-23T08:43:03.991 INFO:teuthology.orchestra.run.smithi059.stdout:create mon.smithi059 on 2024-01-23T08:43:04.205 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-01-23T08:43:04.341 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-01-23T08:43:04.489 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715.target → /etc/systemd/system/ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715.target. 2024-01-23T08:43:04.489 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715.target → /etc/systemd/system/ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715.target. 2024-01-23T08:43:04.649 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@mon.smithi059 2024-01-23T08:43:04.649 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@mon.smithi059.service: Unit ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@mon.smithi059.service not loaded. 2024-01-23T08:43:04.808 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715.target.wants/ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@mon.smithi059.service → /etc/systemd/system/ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@.service. 2024-01-23T08:43:05.316 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:05 smithi059 podman[28031]: 2024-01-23 08:43:05.257266138 +0000 UTC m=+0.326933317 container start 933eaf0f6373d72356e44cfe79808fde8476be515d2bea58c9aa6be4e2dde068 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059, io.buildah.version=1.31.3, org.label-schema.build-date=20240118, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=quincy-b824e5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-01-23T08:43:05.316 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:05 smithi059 ceph-mon[28050]: mkfs 3fc07a38-b9cb-11ee-95b1-87774f69a715 2024-01-23T08:43:05.364 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-01-23T08:43:05.364 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout disabled 2024-01-23T08:43:05.371 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-01-23T08:43:05.372 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout inactive 2024-01-23T08:43:05.372 INFO:teuthology.orchestra.run.smithi059.stdout:firewalld.service is not enabled 2024-01-23T08:43:05.372 INFO:teuthology.orchestra.run.smithi059.stdout:Not possible to enable service . firewalld.service is not available 2024-01-23T08:43:05.372 INFO:teuthology.orchestra.run.smithi059.stdout:Waiting for mon to start... 2024-01-23T08:43:05.372 INFO:teuthology.orchestra.run.smithi059.stdout:Waiting for mon... 2024-01-23T08:43:05.579 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:05 smithi059 ceph-mon[28050]: mon.smithi059 is new leader, mons smithi059 in quorum (ranks 0) 2024-01-23T08:43:05.580 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:05 smithi059 bash[28031]: 933eaf0f6373d72356e44cfe79808fde8476be515d2bea58c9aa6be4e2dde068 2024-01-23T08:43:05.580 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:05 smithi059 systemd[1]: Started Ceph mon.smithi059 for 3fc07a38-b9cb-11ee-95b1-87774f69a715. 2024-01-23T08:43:06.433 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:06 smithi059 ceph-mon[28050]: mon.smithi059 is new leader, mons smithi059 in quorum (ranks 0) 2024-01-23T08:43:06.433 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:06 smithi059 ceph-mon[28050]: monmap e1: 1 mons at {smithi059=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-23T08:43:06.433 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:06 smithi059 ceph-mon[28050]: fsmap 2024-01-23T08:43:06.433 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:06 smithi059 ceph-mon[28050]: osdmap e1: 0 total, 0 up, 0 in 2024-01-23T08:43:06.433 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:06 smithi059 ceph-mon[28050]: mgrmap e1: no daemons active 2024-01-23T08:43:06.433 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:06 smithi059 ceph-mon[28050]: from='client.? 172.21.15.59:0/3490731101' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-01-23T08:43:06.441 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout cluster: 2024-01-23T08:43:06.441 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout id: 3fc07a38-b9cb-11ee-95b1-87774f69a715 2024-01-23T08:43:06.441 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-01-23T08:43:06.442 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-01-23T08:43:06.442 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout services: 2024-01-23T08:43:06.442 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi059 (age 0.551135s) 2024-01-23T08:43:06.442 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-01-23T08:43:06.442 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-01-23T08:43:06.442 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-01-23T08:43:06.442 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout data: 2024-01-23T08:43:06.442 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-01-23T08:43:06.442 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-01-23T08:43:06.442 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-01-23T08:43:06.442 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout pgs: 2024-01-23T08:43:06.442 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-01-23T08:43:06.442 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.787+0000 7f4e5fa15700 1 Processor -- start 2024-01-23T08:43:06.442 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.787+0000 7f4e5fa15700 1 -- start start 2024-01-23T08:43:06.442 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.787+0000 7f4e5fa15700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e58103b40 0x7f4e58103f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:06.442 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.787+0000 7f4e5fa15700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e58104520 con 0x7f4e58103b40 2024-01-23T08:43:06.443 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.787+0000 7f4e5d7b1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e58103b40 0x7f4e58103f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:06.443 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.787+0000 7f4e5d7b1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e58103b40 0x7f4e58103f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50874/0 (socket says 172.21.15.59:50874) 2024-01-23T08:43:06.443 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.787+0000 7f4e5d7b1700 1 -- 172.21.15.59:0/1338469712 learned_addr learned my addr 172.21.15.59:0/1338469712 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:06.443 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.788+0000 7f4e5d7b1700 1 -- 172.21.15.59:0/1338469712 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e58104660 con 0x7f4e58103b40 2024-01-23T08:43:06.443 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.788+0000 7f4e5d7b1700 1 --2- 172.21.15.59:0/1338469712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e58103b40 0x7f4e58103f50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4e48009e50 tx=0x7f4e48009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=182b0ca583eab2c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:06.443 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.788+0000 7f4e57fff700 1 -- 172.21.15.59:0/1338469712 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e480092e0 con 0x7f4e58103b40 2024-01-23T08:43:06.443 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.788+0000 7f4e57fff700 1 -- 172.21.15.59:0/1338469712 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4800ba20 con 0x7f4e58103b40 2024-01-23T08:43:06.443 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.788+0000 7f4e57fff700 1 -- 172.21.15.59:0/1338469712 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e4800bd30 con 0x7f4e58103b40 2024-01-23T08:43:06.443 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.788+0000 7f4e5fa15700 1 -- 172.21.15.59:0/1338469712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e58103b40 msgr2=0x7f4e58103f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:06.443 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.789+0000 7f4e5fa15700 1 --2- 172.21.15.59:0/1338469712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e58103b40 0x7f4e58103f50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4e48009e50 tx=0x7f4e48009840 comp rx=0 tx=0).stop 2024-01-23T08:43:06.443 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.789+0000 7f4e5fa15700 1 -- 172.21.15.59:0/1338469712 shutdown_connections 2024-01-23T08:43:06.443 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.789+0000 7f4e5fa15700 1 --2- 172.21.15.59:0/1338469712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e58103b40 0x7f4e58103f50 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:06.443 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.789+0000 7f4e5fa15700 1 -- 172.21.15.59:0/1338469712 >> 172.21.15.59:0/1338469712 conn(0x7f4e580fc260 msgr2=0x7f4e580fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:06.443 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.789+0000 7f4e5fa15700 1 -- 172.21.15.59:0/1338469712 shutdown_connections 2024-01-23T08:43:06.443 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.789+0000 7f4e5fa15700 1 -- 172.21.15.59:0/1338469712 wait complete. 2024-01-23T08:43:06.444 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.789+0000 7f4e5fa15700 1 Processor -- start 2024-01-23T08:43:06.444 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.789+0000 7f4e5fa15700 1 -- start start 2024-01-23T08:43:06.444 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.789+0000 7f4e5fa15700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e58103b40 0x7f4e581031a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:06.444 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.789+0000 7f4e5fa15700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e581036e0 con 0x7f4e58103b40 2024-01-23T08:43:06.444 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.790+0000 7f4e5d7b1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e58103b40 0x7f4e581031a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:06.444 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.790+0000 7f4e5d7b1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e58103b40 0x7f4e581031a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50878/0 (socket says 172.21.15.59:50878) 2024-01-23T08:43:06.444 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.790+0000 7f4e5d7b1700 1 -- 172.21.15.59:0/3490731101 learned_addr learned my addr 172.21.15.59:0/3490731101 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:06.444 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.790+0000 7f4e5d7b1700 1 -- 172.21.15.59:0/3490731101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e48009ae0 con 0x7f4e58103b40 2024-01-23T08:43:06.444 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.790+0000 7f4e5d7b1700 1 --2- 172.21.15.59:0/3490731101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e58103b40 0x7f4e581031a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4e48003000 tx=0x7f4e48017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:06.444 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.790+0000 7f4e55ffb700 1 -- 172.21.15.59:0/3490731101 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e48012b90 con 0x7f4e58103b40 2024-01-23T08:43:06.444 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.790+0000 7f4e55ffb700 1 -- 172.21.15.59:0/3490731101 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e48012d10 con 0x7f4e58103b40 2024-01-23T08:43:06.444 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.790+0000 7f4e55ffb700 1 -- 172.21.15.59:0/3490731101 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e4801e400 con 0x7f4e58103b40 2024-01-23T08:43:06.444 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.790+0000 7f4e5fa15700 1 -- 172.21.15.59:0/3490731101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e58101800 con 0x7f4e58103b40 2024-01-23T08:43:06.444 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.790+0000 7f4e5fa15700 1 -- 172.21.15.59:0/3490731101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e58101c70 con 0x7f4e58103b40 2024-01-23T08:43:06.444 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.791+0000 7f4e55ffb700 1 -- 172.21.15.59:0/3490731101 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f4e48019030 con 0x7f4e58103b40 2024-01-23T08:43:06.445 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.791+0000 7f4e55ffb700 1 -- 172.21.15.59:0/3490731101 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f4e4801ed00 con 0x7f4e58103b40 2024-01-23T08:43:06.445 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.791+0000 7f4e5fa15700 1 -- 172.21.15.59:0/3490731101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e5805f450 con 0x7f4e58103b40 2024-01-23T08:43:06.445 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.794+0000 7f4e55ffb700 1 -- 172.21.15.59:0/3490731101 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f4e48017070 con 0x7f4e58103b40 2024-01-23T08:43:06.445 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.865+0000 7f4e5fa15700 1 -- 172.21.15.59:0/3490731101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f4e5810a080 con 0x7f4e58103b40 2024-01-23T08:43:06.445 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.865+0000 7f4e55ffb700 1 -- 172.21.15.59:0/3490731101 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f4e4801e6d0 con 0x7f4e58103b40 2024-01-23T08:43:06.445 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.866+0000 7f4e5fa15700 1 -- 172.21.15.59:0/3490731101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e58103b40 msgr2=0x7f4e581031a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:06.445 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.866+0000 7f4e5fa15700 1 --2- 172.21.15.59:0/3490731101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e58103b40 0x7f4e581031a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4e48003000 tx=0x7f4e48017000 comp rx=0 tx=0).stop 2024-01-23T08:43:06.445 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.866+0000 7f4e5fa15700 1 -- 172.21.15.59:0/3490731101 shutdown_connections 2024-01-23T08:43:06.445 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.866+0000 7f4e5fa15700 1 --2- 172.21.15.59:0/3490731101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4e58103b40 0x7f4e581031a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:06.445 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.866+0000 7f4e5fa15700 1 -- 172.21.15.59:0/3490731101 >> 172.21.15.59:0/3490731101 conn(0x7f4e580fc260 msgr2=0x7f4e581053a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:06.445 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.866+0000 7f4e5fa15700 1 -- 172.21.15.59:0/3490731101 shutdown_connections 2024-01-23T08:43:06.445 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:05.866+0000 7f4e5fa15700 1 -- 172.21.15.59:0/3490731101 wait complete. 2024-01-23T08:43:06.445 INFO:teuthology.orchestra.run.smithi059.stdout:mon is available 2024-01-23T08:43:06.445 INFO:teuthology.orchestra.run.smithi059.stdout:Assimilating anything we can from ceph.conf... 2024-01-23T08:43:07.530 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:07 smithi059 ceph-mon[28050]: from='client.? 172.21.15.59:0/849640239' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-23T08:43:07.530 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:07 smithi059 ceph-mon[28050]: from='client.? 172.21.15.59:0/849640239' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-23T08:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-01-23T08:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout [global] 2024-01-23T08:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout fsid = 3fc07a38-b9cb-11ee-95b1-87774f69a715 2024-01-23T08:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.59:3300,v1:172.21.15.59:6789] 2024-01-23T08:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-01-23T08:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-01-23T08:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-01-23T08:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-01-23T08:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-01-23T08:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout [mgr] 2024-01-23T08:43:07.539 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-01-23T08:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-01-23T08:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout [osd] 2024-01-23T08:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-01-23T08:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-01-23T08:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-01-23T08:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-01-23T08:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-01-23T08:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.949+0000 7ff8555f1700 1 Processor -- start 2024-01-23T08:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.949+0000 7ff8555f1700 1 -- start start 2024-01-23T08:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.950+0000 7ff8555f1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff850103030 0x7ff850105410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.950+0000 7ff8555f1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff850076db0 con 0x7ff850103030 2024-01-23T08:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.950+0000 7ff84effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff850103030 0x7ff850105410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.950+0000 7ff84effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff850103030 0x7ff850105410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50892/0 (socket says 172.21.15.59:50892) 2024-01-23T08:43:07.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.950+0000 7ff84effd700 1 -- 172.21.15.59:0/1254877147 learned_addr learned my addr 172.21.15.59:0/1254877147 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.950+0000 7ff84effd700 1 -- 172.21.15.59:0/1254877147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff850076ef0 con 0x7ff850103030 2024-01-23T08:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.950+0000 7ff84effd700 1 --2- 172.21.15.59:0/1254877147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff850103030 0x7ff850105410 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff848009e50 tx=0x7ff848009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b64518380b69abb server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.951+0000 7ff84dffb700 1 -- 172.21.15.59:0/1254877147 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8480092e0 con 0x7ff850103030 2024-01-23T08:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.951+0000 7ff84dffb700 1 -- 172.21.15.59:0/1254877147 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84800ba20 con 0x7ff850103030 2024-01-23T08:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.951+0000 7ff84dffb700 1 -- 172.21.15.59:0/1254877147 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff84800bd30 con 0x7ff850103030 2024-01-23T08:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.951+0000 7ff8555f1700 1 -- 172.21.15.59:0/1254877147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff850103030 msgr2=0x7ff850105410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.951+0000 7ff8555f1700 1 --2- 172.21.15.59:0/1254877147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff850103030 0x7ff850105410 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff848009e50 tx=0x7ff848009840 comp rx=0 tx=0).stop 2024-01-23T08:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.951+0000 7ff8555f1700 1 -- 172.21.15.59:0/1254877147 shutdown_connections 2024-01-23T08:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.951+0000 7ff8555f1700 1 --2- 172.21.15.59:0/1254877147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff850103030 0x7ff850105410 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.951+0000 7ff8555f1700 1 -- 172.21.15.59:0/1254877147 >> 172.21.15.59:0/1254877147 conn(0x7ff8500fcad0 msgr2=0x7ff8500fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.951+0000 7ff8555f1700 1 -- 172.21.15.59:0/1254877147 shutdown_connections 2024-01-23T08:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.951+0000 7ff8555f1700 1 -- 172.21.15.59:0/1254877147 wait complete. 2024-01-23T08:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.952+0000 7ff8555f1700 1 Processor -- start 2024-01-23T08:43:07.541 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.952+0000 7ff8555f1700 1 -- start start 2024-01-23T08:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.952+0000 7ff8555f1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff850103030 0x7ff8501028e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.952+0000 7ff8555f1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff850102e20 con 0x7ff850103030 2024-01-23T08:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.952+0000 7ff84effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff850103030 0x7ff8501028e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.952+0000 7ff84effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff850103030 0x7ff8501028e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50906/0 (socket says 172.21.15.59:50906) 2024-01-23T08:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.952+0000 7ff84effd700 1 -- 172.21.15.59:0/849640239 learned_addr learned my addr 172.21.15.59:0/849640239 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.952+0000 7ff84effd700 1 -- 172.21.15.59:0/849640239 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff848009ae0 con 0x7ff850103030 2024-01-23T08:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.953+0000 7ff84effd700 1 --2- 172.21.15.59:0/849640239 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff850103030 0x7ff8501028e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff850076200 tx=0x7ff848017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.953+0000 7ff8477fe700 1 -- 172.21.15.59:0/849640239 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff848012b90 con 0x7ff850103030 2024-01-23T08:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.953+0000 7ff8477fe700 1 -- 172.21.15.59:0/849640239 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff848012d10 con 0x7ff850103030 2024-01-23T08:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.953+0000 7ff8555f1700 1 -- 172.21.15.59:0/849640239 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff850100f40 con 0x7ff850103030 2024-01-23T08:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.953+0000 7ff8477fe700 1 -- 172.21.15.59:0/849640239 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff84801e400 con 0x7ff850103030 2024-01-23T08:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.953+0000 7ff8555f1700 1 -- 172.21.15.59:0/849640239 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8501013b0 con 0x7ff850103030 2024-01-23T08:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.953+0000 7ff8477fe700 1 -- 172.21.15.59:0/849640239 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7ff848019030 con 0x7ff850103030 2024-01-23T08:43:07.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.953+0000 7ff8477fe700 1 -- 172.21.15.59:0/849640239 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff84801ed00 con 0x7ff850103030 2024-01-23T08:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.954+0000 7ff8555f1700 1 -- 172.21.15.59:0/849640239 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff85005f450 con 0x7ff850103030 2024-01-23T08:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:06.956+0000 7ff8477fe700 1 -- 172.21.15.59:0/849640239 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7ff850076ef0 con 0x7ff850103030 2024-01-23T08:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.026+0000 7ff8555f1700 1 -- 172.21.15.59:0/849640239 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7ff850105b30 con 0x7ff850103030 2024-01-23T08:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.029+0000 7ff8477fe700 1 -- 172.21.15.59:0/849640239 <== mon.0 v2:172.21.15.59:3300/0 7 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7ff848039450 con 0x7ff850103030 2024-01-23T08:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.029+0000 7ff8477fe700 1 -- 172.21.15.59:0/849640239 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7ff848017070 con 0x7ff850103030 2024-01-23T08:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.030+0000 7ff8555f1700 1 -- 172.21.15.59:0/849640239 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff850103030 msgr2=0x7ff8501028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.030+0000 7ff8555f1700 1 --2- 172.21.15.59:0/849640239 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff850103030 0x7ff8501028e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff850076200 tx=0x7ff848017000 comp rx=0 tx=0).stop 2024-01-23T08:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.030+0000 7ff8555f1700 1 -- 172.21.15.59:0/849640239 shutdown_connections 2024-01-23T08:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.030+0000 7ff8555f1700 1 --2- 172.21.15.59:0/849640239 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff850103030 0x7ff8501028e0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.030+0000 7ff8555f1700 1 -- 172.21.15.59:0/849640239 >> 172.21.15.59:0/849640239 conn(0x7ff8500fcad0 msgr2=0x7ff8500fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.030+0000 7ff8555f1700 1 -- 172.21.15.59:0/849640239 shutdown_connections 2024-01-23T08:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.030+0000 7ff8555f1700 1 -- 172.21.15.59:0/849640239 wait complete. 2024-01-23T08:43:07.543 INFO:teuthology.orchestra.run.smithi059.stdout:Generating new minimal ceph.conf... 2024-01-23T08:43:08.485 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.939+0000 7fe6271a3700 1 Processor -- start 2024-01-23T08:43:08.485 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.939+0000 7fe6271a3700 1 -- start start 2024-01-23T08:43:08.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.939+0000 7fe6271a3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe620102ae0 0x7fe620102ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:08.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.939+0000 7fe6271a3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe620103430 con 0x7fe620102ae0 2024-01-23T08:43:08.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.940+0000 7fe624f3f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe620102ae0 0x7fe620102ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:08.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.940+0000 7fe624f3f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe620102ae0 0x7fe620102ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60020/0 (socket says 172.21.15.59:60020) 2024-01-23T08:43:08.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.940+0000 7fe624f3f700 1 -- 172.21.15.59:0/3743584116 learned_addr learned my addr 172.21.15.59:0/3743584116 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:08.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.940+0000 7fe624f3f700 1 -- 172.21.15.59:0/3743584116 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe620103570 con 0x7fe620102ae0 2024-01-23T08:43:08.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.940+0000 7fe624f3f700 1 --2- 172.21.15.59:0/3743584116 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe620102ae0 0x7fe620102ef0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe618009e50 tx=0x7fe618009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff7f3fca7870e83f server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:08.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.940+0000 7fe61f7fe700 1 -- 172.21.15.59:0/3743584116 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6180092e0 con 0x7fe620102ae0 2024-01-23T08:43:08.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.940+0000 7fe61f7fe700 1 -- 172.21.15.59:0/3743584116 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe618012430 con 0x7fe620102ae0 2024-01-23T08:43:08.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.940+0000 7fe61f7fe700 1 -- 172.21.15.59:0/3743584116 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6180075c0 con 0x7fe620102ae0 2024-01-23T08:43:08.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.941+0000 7fe6271a3700 1 -- 172.21.15.59:0/3743584116 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe620102ae0 msgr2=0x7fe620102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:08.486 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.941+0000 7fe6271a3700 1 --2- 172.21.15.59:0/3743584116 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe620102ae0 0x7fe620102ef0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe618009e50 tx=0x7fe618009840 comp rx=0 tx=0).stop 2024-01-23T08:43:08.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.941+0000 7fe6271a3700 1 -- 172.21.15.59:0/3743584116 shutdown_connections 2024-01-23T08:43:08.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.941+0000 7fe6271a3700 1 --2- 172.21.15.59:0/3743584116 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe620102ae0 0x7fe620102ef0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:08.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.941+0000 7fe6271a3700 1 -- 172.21.15.59:0/3743584116 >> 172.21.15.59:0/3743584116 conn(0x7fe6200fc2d0 msgr2=0x7fe6200fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:08.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.941+0000 7fe6271a3700 1 -- 172.21.15.59:0/3743584116 shutdown_connections 2024-01-23T08:43:08.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.941+0000 7fe6271a3700 1 -- 172.21.15.59:0/3743584116 wait complete. 2024-01-23T08:43:08.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.941+0000 7fe6271a3700 1 Processor -- start 2024-01-23T08:43:08.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.941+0000 7fe6271a3700 1 -- start start 2024-01-23T08:43:08.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.942+0000 7fe6271a3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe620102ae0 0x7fe6201133d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:08.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.942+0000 7fe6271a3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe620113910 con 0x7fe620102ae0 2024-01-23T08:43:08.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.942+0000 7fe624f3f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe620102ae0 0x7fe6201133d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:08.487 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.942+0000 7fe624f3f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe620102ae0 0x7fe6201133d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60030/0 (socket says 172.21.15.59:60030) 2024-01-23T08:43:08.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.942+0000 7fe624f3f700 1 -- 172.21.15.59:0/144149376 learned_addr learned my addr 172.21.15.59:0/144149376 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:08.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.942+0000 7fe624f3f700 1 -- 172.21.15.59:0/144149376 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe618009ae0 con 0x7fe620102ae0 2024-01-23T08:43:08.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.942+0000 7fe624f3f700 1 --2- 172.21.15.59:0/144149376 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe620102ae0 0x7fe6201133d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe6180127d0 tx=0x7fe618009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:08.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.942+0000 7fe61d7fa700 1 -- 172.21.15.59:0/144149376 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6180092e0 con 0x7fe620102ae0 2024-01-23T08:43:08.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.942+0000 7fe61d7fa700 1 -- 172.21.15.59:0/144149376 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe618007cf0 con 0x7fe620102ae0 2024-01-23T08:43:08.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.942+0000 7fe61d7fa700 1 -- 172.21.15.59:0/144149376 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe61801c970 con 0x7fe620102ae0 2024-01-23T08:43:08.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.943+0000 7fe6271a3700 1 -- 172.21.15.59:0/144149376 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe620113b10 con 0x7fe620102ae0 2024-01-23T08:43:08.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.943+0000 7fe6271a3700 1 -- 172.21.15.59:0/144149376 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe620113f80 con 0x7fe620102ae0 2024-01-23T08:43:08.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.943+0000 7fe61d7fa700 1 -- 172.21.15.59:0/144149376 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fe618019070 con 0x7fe620102ae0 2024-01-23T08:43:08.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.943+0000 7fe61d7fa700 1 -- 172.21.15.59:0/144149376 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe618025b40 con 0x7fe620102ae0 2024-01-23T08:43:08.488 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.944+0000 7fe6271a3700 1 -- 172.21.15.59:0/144149376 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe620108720 con 0x7fe620102ae0 2024-01-23T08:43:08.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:07.946+0000 7fe61d7fa700 1 -- 172.21.15.59:0/144149376 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fe61802a060 con 0x7fe620102ae0 2024-01-23T08:43:08.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:08.014+0000 7fe6271a3700 1 -- 172.21.15.59:0/144149376 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fe620116980 con 0x7fe620102ae0 2024-01-23T08:43:08.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:08.015+0000 7fe61d7fa700 1 -- 172.21.15.59:0/144149376 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fe618020070 con 0x7fe620102ae0 2024-01-23T08:43:08.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:08.015+0000 7fe6271a3700 1 -- 172.21.15.59:0/144149376 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe620102ae0 msgr2=0x7fe6201133d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:08.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:08.015+0000 7fe6271a3700 1 --2- 172.21.15.59:0/144149376 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe620102ae0 0x7fe6201133d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe6180127d0 tx=0x7fe618009840 comp rx=0 tx=0).stop 2024-01-23T08:43:08.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:08.015+0000 7fe6271a3700 1 -- 172.21.15.59:0/144149376 shutdown_connections 2024-01-23T08:43:08.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:08.015+0000 7fe6271a3700 1 --2- 172.21.15.59:0/144149376 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe620102ae0 0x7fe6201133d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:08.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:08.015+0000 7fe6271a3700 1 -- 172.21.15.59:0/144149376 >> 172.21.15.59:0/144149376 conn(0x7fe6200fc2d0 msgr2=0x7fe6200fd9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:08.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:08.015+0000 7fe6271a3700 1 -- 172.21.15.59:0/144149376 shutdown_connections 2024-01-23T08:43:08.489 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:08.015+0000 7fe6271a3700 1 -- 172.21.15.59:0/144149376 wait complete. 2024-01-23T08:43:08.489 INFO:teuthology.orchestra.run.smithi059.stdout:Restarting the monitor... 2024-01-23T08:43:08.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:08 smithi059 ceph-mon[28050]: from='client.? 172.21.15.59:0/144149376' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:43:08.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:08 smithi059 systemd[1]: Stopping Ceph mon.smithi059 for 3fc07a38-b9cb-11ee-95b1-87774f69a715... 2024-01-23T08:43:09.040 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28046]: 2024-01-23T08:43:08.724+0000 7f5d2b707700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi059 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-01-23T08:43:09.040 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28046]: 2024-01-23T08:43:08.724+0000 7f5d2b707700 -1 mon.smithi059@0(leader) e1 *** Got Signal Terminated *** 2024-01-23T08:43:09.040 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:08 smithi059 podman[28296]: 2024-01-23 08:43:08.797217414 +0000 UTC m=+0.154659133 container died 933eaf0f6373d72356e44cfe79808fde8476be515d2bea58c9aa6be4e2dde068 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059, io.buildah.version=1.31.3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=quincy-b824e5b, org.label-schema.build-date=20240118, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-01-23T08:43:09.415 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 podman[28296]: 2024-01-23 08:43:09.14218153 +0000 UTC m=+0.499623245 container cleanup 933eaf0f6373d72356e44cfe79808fde8476be515d2bea58c9aa6be4e2dde068 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059, GIT_CLEAN=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, io.buildah.version=1.31.3, org.label-schema.license=GPLv2, RELEASE=quincy-b824e5b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240118) 2024-01-23T08:43:09.415 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 bash[28296]: ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059 2024-01-23T08:43:09.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 podman[28310]: 2024-01-23 08:43:09.418130859 +0000 UTC m=+0.622799350 container remove 933eaf0f6373d72356e44cfe79808fde8476be515d2bea58c9aa6be4e2dde068 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240118, io.buildah.version=1.31.3, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-b824e5b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-01-23T08:43:09.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 systemd[1]: ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@mon.smithi059.service: Deactivated successfully. 2024-01-23T08:43:09.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 systemd[1]: Stopped Ceph mon.smithi059 for 3fc07a38-b9cb-11ee-95b1-87774f69a715. 2024-01-23T08:43:09.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 systemd[1]: Starting Ceph mon.smithi059 for 3fc07a38-b9cb-11ee-95b1-87774f69a715... 2024-01-23T08:43:10.010 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 podman[28405]: 2024-01-23 08:43:09.752356117 +0000 UTC m=+0.094667707 container create 810a40012e10c7bcb0c50786b86e91f99951825a767752b4654b71ec2df613f0 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240118, GIT_CLEAN=True, RELEASE=quincy-b824e5b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-01-23T08:43:10.010 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 podman[28405]: 2024-01-23 08:43:09.67494498 +0000 UTC m=+0.017256572 image pull 61b360e6ff650187c031550cf8bb1c0d3e28fc466cffb29cb06fa0da2544d680 quay.ceph.io/ceph-ci/ceph:quincy 2024-01-23T08:43:10.010 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 podman[28405]: 2024-01-23 08:43:09.92964385 +0000 UTC m=+0.271955452 container init 810a40012e10c7bcb0c50786b86e91f99951825a767752b4654b71ec2df613f0 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, io.buildah.version=1.31.3, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240118, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=quincy-b824e5b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-01-23T08:43:10.010 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 podman[28405]: 2024-01-23 08:43:09.932642679 +0000 UTC m=+0.274954271 container start 810a40012e10c7bcb0c50786b86e91f99951825a767752b4654b71ec2df613f0 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059, org.label-schema.license=GPLv2, org.label-schema.build-date=20240118, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.31.3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=quincy-b824e5b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e) 2024-01-23T08:43:10.010 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: set uid:gid to 167:167 (ceph:ceph) 2024-01-23T08:43:10.011 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: ceph version 17.2.7-397-gb824e5be (b824e5befce1d427fda2bc90ae52665f0c7e2770) quincy (stable), process ceph-mon, pid 2 2024-01-23T08:43:10.011 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: pidfile_write: ignore empty --pid-file 2024-01-23T08:43:10.011 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: load: jerasure load: lrc 2024-01-23T08:43:10.011 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: RocksDB version: 6.15.5 2024-01-23T08:43:10.011 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-23T08:43:10.011 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Compile date Jan 22 2024 2024-01-23T08:43:10.011 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: DB SUMMARY 2024-01-23T08:43:10.011 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: DB Session ID: ATAJOADU756EPV9F1JQD 2024-01-23T08:43:10.011 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: CURRENT file: CURRENT 2024-01-23T08:43:10.011 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: IDENTITY file: IDENTITY 2024-01-23T08:43:10.011 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-01-23T08:43:10.011 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi059/store.db dir, Total Num: 1, files: 000008.sst 2024-01-23T08:43:10.012 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi059/store.db: 000010.log size: 96752 ; 2024-01-23T08:43:10.012 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.error_if_exists: 0 2024-01-23T08:43:10.012 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.create_if_missing: 0 2024-01-23T08:43:10.012 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.paranoid_checks: 1 2024-01-23T08:43:10.012 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-01-23T08:43:10.012 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.env: 0x5559d075aa80 2024-01-23T08:43:10.012 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.fs: Posix File System 2024-01-23T08:43:10.012 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.info_log: 0x5559d2b43de0 2024-01-23T08:43:10.013 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_file_opening_threads: 16 2024-01-23T08:43:10.013 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.statistics: (nil) 2024-01-23T08:43:10.013 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.use_fsync: 0 2024-01-23T08:43:10.013 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_log_file_size: 0 2024-01-23T08:43:10.013 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-23T08:43:10.013 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.log_file_time_to_roll: 0 2024-01-23T08:43:10.013 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.keep_log_file_num: 1000 2024-01-23T08:43:10.013 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.recycle_log_file_num: 0 2024-01-23T08:43:10.013 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.allow_fallocate: 1 2024-01-23T08:43:10.013 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.allow_mmap_reads: 0 2024-01-23T08:43:10.013 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.allow_mmap_writes: 0 2024-01-23T08:43:10.014 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.use_direct_reads: 0 2024-01-23T08:43:10.014 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-23T08:43:10.014 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.create_missing_column_families: 0 2024-01-23T08:43:10.014 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.db_log_dir: 2024-01-23T08:43:10.014 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi059/store.db 2024-01-23T08:43:10.014 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.table_cache_numshardbits: 6 2024-01-23T08:43:10.014 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.WAL_ttl_seconds: 0 2024-01-23T08:43:10.014 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.WAL_size_limit_MB: 0 2024-01-23T08:43:10.014 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-23T08:43:10.014 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-23T08:43:10.014 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.is_fd_close_on_exec: 1 2024-01-23T08:43:10.014 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.advise_random_on_open: 1 2024-01-23T08:43:10.014 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.db_write_buffer_size: 0 2024-01-23T08:43:10.014 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.write_buffer_manager: 0x5559d38b8a80 2024-01-23T08:43:10.014 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-23T08:43:10.015 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-23T08:43:10.015 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-23T08:43:10.015 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.use_adaptive_mutex: 0 2024-01-23T08:43:10.015 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.rate_limiter: (nil) 2024-01-23T08:43:10.015 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-23T08:43:10.015 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.wal_recovery_mode: 2 2024-01-23T08:43:10.015 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.enable_thread_tracking: 0 2024-01-23T08:43:10.015 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.enable_pipelined_write: 0 2024-01-23T08:43:10.016 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.unordered_write: 0 2024-01-23T08:43:10.017 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-23T08:43:10.017 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-23T08:43:10.017 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-23T08:43:10.017 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-23T08:43:10.017 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.row_cache: None 2024-01-23T08:43:10.017 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.wal_filter: None 2024-01-23T08:43:10.017 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-23T08:43:10.017 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.allow_ingest_behind: 0 2024-01-23T08:43:10.017 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.preserve_deletes: 0 2024-01-23T08:43:10.017 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.two_write_queues: 0 2024-01-23T08:43:10.017 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.manual_wal_flush: 0 2024-01-23T08:43:10.017 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.atomic_flush: 0 2024-01-23T08:43:10.017 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-23T08:43:10.018 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.persist_stats_to_disk: 0 2024-01-23T08:43:10.018 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.write_dbid_to_manifest: 0 2024-01-23T08:43:10.018 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.log_readahead_size: 0 2024-01-23T08:43:10.018 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-01-23T08:43:10.018 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.best_efforts_recovery: 0 2024-01-23T08:43:10.018 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-01-23T08:43:10.018 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-01-23T08:43:10.018 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.allow_data_in_errors: 0 2024-01-23T08:43:10.018 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.db_host_id: __hostname__ 2024-01-23T08:43:10.018 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_background_jobs: 2 2024-01-23T08:43:10.018 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_background_compactions: -1 2024-01-23T08:43:10.018 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_subcompactions: 1 2024-01-23T08:43:10.018 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-23T08:43:10.018 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-23T08:43:10.018 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.delayed_write_rate : 16777216 2024-01-23T08:43:10.019 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_total_wal_size: 0 2024-01-23T08:43:10.019 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-23T08:43:10.019 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.stats_dump_period_sec: 600 2024-01-23T08:43:10.019 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.stats_persist_period_sec: 600 2024-01-23T08:43:10.019 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-23T08:43:10.019 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_open_files: -1 2024-01-23T08:43:10.019 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.bytes_per_sync: 0 2024-01-23T08:43:10.019 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.wal_bytes_per_sync: 0 2024-01-23T08:43:10.019 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.strict_bytes_per_sync: 0 2024-01-23T08:43:10.019 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.compaction_readahead_size: 0 2024-01-23T08:43:10.020 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_background_flushes: -1 2024-01-23T08:43:10.020 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Compression algorithms supported: 2024-01-23T08:43:10.020 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-23T08:43:10.020 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: kZSTD supported: 0 2024-01-23T08:43:10.020 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: kXpressCompression supported: 0 2024-01-23T08:43:10.020 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: kLZ4HCCompression supported: 1 2024-01-23T08:43:10.020 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: kZlibCompression supported: 1 2024-01-23T08:43:10.020 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: kSnappyCompression supported: 1 2024-01-23T08:43:10.020 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: kLZ4Compression supported: 1 2024-01-23T08:43:10.021 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: kBZip2Compression supported: 0 2024-01-23T08:43:10.021 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Fast CRC32 supported: Supported on x86 2024-01-23T08:43:10.021 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi059/store.db/MANIFEST-000009 2024-01-23T08:43:10.021 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-01-23T08:43:10.021 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-23T08:43:10.021 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.merge_operator: 2024-01-23T08:43:10.021 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.compaction_filter: None 2024-01-23T08:43:10.021 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.compaction_filter_factory: None 2024-01-23T08:43:10.021 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.sst_partitioner_factory: None 2024-01-23T08:43:10.021 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.memtable_factory: SkipListFactory 2024-01-23T08:43:10.021 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.table_factory: BlockBasedTable 2024-01-23T08:43:10.021 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5559d2b98fd8) 2024-01-23T08:43:10.022 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: cache_index_and_filter_blocks: 1 2024-01-23T08:43:10.022 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-23T08:43:10.022 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-23T08:43:10.022 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: pin_top_level_index_and_filter: 1 2024-01-23T08:43:10.022 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: index_type: 0 2024-01-23T08:43:10.022 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: data_block_index_type: 0 2024-01-23T08:43:10.022 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: index_shortening: 1 2024-01-23T08:43:10.022 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: data_block_hash_table_util_ratio: 0.750000 2024-01-23T08:43:10.022 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: hash_index_allow_collision: 1 2024-01-23T08:43:10.022 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: checksum: 1 2024-01-23T08:43:10.022 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: no_block_cache: 0 2024-01-23T08:43:10.022 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: block_cache: 0x5559d2bcb090 2024-01-23T08:43:10.022 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: block_cache_name: BinnedLRUCache 2024-01-23T08:43:10.022 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: block_cache_options: 2024-01-23T08:43:10.023 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: capacity : 536870912 2024-01-23T08:43:10.023 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: num_shard_bits : 4 2024-01-23T08:43:10.023 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: strict_capacity_limit : 0 2024-01-23T08:43:10.023 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: high_pri_pool_ratio: 0.000 2024-01-23T08:43:10.023 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: block_cache_compressed: (nil) 2024-01-23T08:43:10.023 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: persistent_cache: (nil) 2024-01-23T08:43:10.023 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: block_size: 4096 2024-01-23T08:43:10.023 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: block_size_deviation: 10 2024-01-23T08:43:10.023 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: block_restart_interval: 16 2024-01-23T08:43:10.023 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: index_block_restart_interval: 1 2024-01-23T08:43:10.023 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: metadata_block_size: 4096 2024-01-23T08:43:10.023 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: partition_filters: 0 2024-01-23T08:43:10.023 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: use_delta_encoding: 1 2024-01-23T08:43:10.023 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-23T08:43:10.023 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: whole_key_filtering: 1 2024-01-23T08:43:10.024 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: verify_compression: 0 2024-01-23T08:43:10.024 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: read_amp_bytes_per_bit: 0 2024-01-23T08:43:10.024 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: format_version: 4 2024-01-23T08:43:10.024 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: enable_index_compression: 1 2024-01-23T08:43:10.024 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: block_align: 0 2024-01-23T08:43:10.024 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.write_buffer_size: 33554432 2024-01-23T08:43:10.024 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_write_buffer_number: 2 2024-01-23T08:43:10.024 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.compression: NoCompression 2024-01-23T08:43:10.024 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.bottommost_compression: Disabled 2024-01-23T08:43:10.024 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.prefix_extractor: nullptr 2024-01-23T08:43:10.024 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-23T08:43:10.025 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.num_levels: 7 2024-01-23T08:43:10.025 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-23T08:43:10.025 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-23T08:43:10.025 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-23T08:43:10.025 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-23T08:43:10.025 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-23T08:43:10.025 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-23T08:43:10.025 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-23T08:43:10.025 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-23T08:43:10.025 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-01-23T08:43:10.025 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-23T08:43:10.025 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.compression_opts.window_bits: -14 2024-01-23T08:43:10.025 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.compression_opts.level: 32767 2024-01-23T08:43:10.025 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.compression_opts.strategy: 0 2024-01-23T08:43:10.026 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-23T08:43:10.026 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-23T08:43:10.026 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-01-23T08:43:10.026 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.compression_opts.enabled: false 2024-01-23T08:43:10.026 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-23T08:43:10.026 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-23T08:43:10.026 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-23T08:43:10.026 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.target_file_size_base: 67108864 2024-01-23T08:43:10.026 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.target_file_size_multiplier: 1 2024-01-23T08:43:10.026 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-23T08:43:10.026 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-23T08:43:10.026 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-23T08:43:10.026 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-23T08:43:10.026 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-23T08:43:10.027 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-23T08:43:10.027 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-23T08:43:10.027 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-23T08:43:10.027 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-23T08:43:10.027 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-23T08:43:10.027 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-23T08:43:10.027 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-23T08:43:10.027 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.arena_block_size: 4194304 2024-01-23T08:43:10.027 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-23T08:43:10.027 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-23T08:43:10.027 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-23T08:43:10.027 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.disable_auto_compactions: 0 2024-01-23T08:43:10.027 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-23T08:43:10.028 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-23T08:43:10.028 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-23T08:43:10.028 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-23T08:43:10.028 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-23T08:43:10.028 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-23T08:43:10.028 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-23T08:43:10.028 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-23T08:43:10.028 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-23T08:43:10.028 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-23T08:43:10.028 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.table_properties_collectors: 2024-01-23T08:43:10.028 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.inplace_update_support: 0 2024-01-23T08:43:10.028 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.inplace_update_num_locks: 10000 2024-01-23T08:43:10.028 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-23T08:43:10.028 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-23T08:43:10.028 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.memtable_huge_page_size: 0 2024-01-23T08:43:10.029 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.bloom_locality: 0 2024-01-23T08:43:10.029 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.max_successive_merges: 0 2024-01-23T08:43:10.029 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.optimize_filters_for_hits: 0 2024-01-23T08:43:10.029 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.paranoid_file_checks: 0 2024-01-23T08:43:10.029 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.force_consistency_checks: 1 2024-01-23T08:43:10.029 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.report_bg_io_stats: 0 2024-01-23T08:43:10.029 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.ttl: 2592000 2024-01-23T08:43:10.029 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.periodic_compaction_seconds: 0 2024-01-23T08:43:10.029 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.enable_blob_files: false 2024-01-23T08:43:10.029 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.min_blob_size: 0 2024-01-23T08:43:10.029 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.blob_file_size: 268435456 2024-01-23T08:43:10.029 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.blob_compression_type: NoCompression 2024-01-23T08:43:10.029 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.enable_blob_garbage_collection: false 2024-01-23T08:43:10.029 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-01-23T08:43:10.030 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi059/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-23T08:43:10.030 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-01-23T08:43:10.030 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-01-23T08:43:10.030 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: EVENT_LOG_v1 {"time_micros": 1705999389958131, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-01-23T08:43:10.030 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-01-23T08:43:10.030 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-23T08:43:10.030 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: EVENT_LOG_v1 {"time_micros": 1705999389959318, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 92121, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 90216, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13225, "raw_average_key_size": 49, "raw_value_size": 82955, "raw_average_value_size": 310, "num_data_blocks": 11, "num_entries": 267, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705999389, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "cdfb5008-e21c-4ff0-a594-47dfba4392ac", "db_session_id": "ATAJOADU756EPV9F1JQD"}} 2024-01-23T08:43:10.030 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-01-23T08:43:10.030 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: EVENT_LOG_v1 {"time_micros": 1705999389959773, "job": 1, "event": "recovery_finished"} 2024-01-23T08:43:10.030 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi059/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-01-23T08:43:10.030 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x5559d2b5f180 2024-01-23T08:43:10.031 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: DB pointer 0x5559d2c34000 2024-01-23T08:43:10.031 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-01-23T08:43:10.031 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-01-23T08:43:10.031 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ** DB Stats ** 2024-01-23T08:43:10.031 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-23T08:43:10.031 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-23T08:43:10.031 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-23T08:43:10.031 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-23T08:43:10.031 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-23T08:43:10.031 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-23T08:43:10.031 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-23T08:43:10.031 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: 2024-01-23T08:43:10.031 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ** Compaction Stats [default] ** 2024-01-23T08:43:10.032 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-23T08:43:10.032 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-23T08:43:10.032 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: L0 2/0 91.76 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.3 0.00 0.00 1 0.001 0 0 2024-01-23T08:43:10.032 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Sum 2/0 91.76 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.3 0.00 0.00 1 0.001 0 0 2024-01-23T08:43:10.032 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.3 0.00 0.00 1 0.001 0 0 2024-01-23T08:43:10.032 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: 2024-01-23T08:43:10.032 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ** Compaction Stats [default] ** 2024-01-23T08:43:10.032 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-23T08:43:10.032 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-23T08:43:10.032 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 107.3 0.00 0.00 1 0.001 0 0 2024-01-23T08:43:10.032 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-23T08:43:10.032 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-01-23T08:43:10.032 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-23T08:43:10.033 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-01-23T08:43:10.033 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-01-23T08:43:10.033 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: AddFile(Keys): cumulative 0, interval 0 2024-01-23T08:43:10.033 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Cumulative compaction: 0.00 GB write, 21.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-23T08:43:10.033 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Interval compaction: 0.00 GB write, 21.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-23T08:43:10.033 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-23T08:43:10.033 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: 2024-01-23T08:43:10.033 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ** File Read Latency Histogram By Level [default] ** 2024-01-23T08:43:10.033 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: 2024-01-23T08:43:10.033 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ** Compaction Stats [default] ** 2024-01-23T08:43:10.033 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-23T08:43:10.033 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-23T08:43:10.033 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: L0 2/0 91.76 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.3 0.00 0.00 1 0.001 0 0 2024-01-23T08:43:10.033 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Sum 2/0 91.76 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.3 0.00 0.00 1 0.001 0 0 2024-01-23T08:43:10.033 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-23T08:43:10.034 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: 2024-01-23T08:43:10.034 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ** Compaction Stats [default] ** 2024-01-23T08:43:10.034 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-23T08:43:10.034 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-23T08:43:10.034 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 107.3 0.00 0.00 1 0.001 0 0 2024-01-23T08:43:10.034 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-23T08:43:10.034 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-01-23T08:43:10.034 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-23T08:43:10.034 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-01-23T08:43:10.034 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-01-23T08:43:10.034 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: AddFile(Keys): cumulative 0, interval 0 2024-01-23T08:43:10.034 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Cumulative compaction: 0.00 GB write, 20.85 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-23T08:43:10.034 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-23T08:43:10.034 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-23T08:43:10.035 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: 2024-01-23T08:43:10.035 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ** File Read Latency Histogram By Level [default] ** 2024-01-23T08:43:10.035 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: starting mon.smithi059 rank 0 at public addrs [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] at bind addrs [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi059 fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 2024-01-23T08:43:10.035 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: mon.smithi059@-1(???) e1 preinit fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 2024-01-23T08:43:10.035 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: mon.smithi059@-1(???).mds e1 new map 2024-01-23T08:43:10.035 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: mon.smithi059@-1(???).mds e1 print_map 2024-01-23T08:43:10.035 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: e1 2024-01-23T08:43:10.035 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-01-23T08:43:10.035 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-23T08:43:10.035 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: legacy client fscid: -1 2024-01-23T08:43:10.035 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: 2024-01-23T08:43:10.035 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: No filesystems configured 2024-01-23T08:43:10.035 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: mon.smithi059@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-23T08:43:10.035 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: mon.smithi059@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-23T08:43:10.036 INFO:teuthology.orchestra.run.smithi059.stdout:Setting mon public_network to 172.21.0.0/20 2024-01-23T08:43:10.036 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: mon.smithi059@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-23T08:43:10.036 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: mon.smithi059@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-23T08:43:10.036 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:09 smithi059 ceph-mon[28424]: mon.smithi059@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-01-23T08:43:10.036 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:10 smithi059 bash[28405]: 810a40012e10c7bcb0c50786b86e91f99951825a767752b4654b71ec2df613f0 2024-01-23T08:43:10.036 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:10 smithi059 systemd[1]: Started Ceph mon.smithi059 for 3fc07a38-b9cb-11ee-95b1-87774f69a715. 2024-01-23T08:43:10.312 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:10 smithi059 ceph-mon[28424]: mon.smithi059 is new leader, mons smithi059 in quorum (ranks 0) 2024-01-23T08:43:10.313 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:10 smithi059 ceph-mon[28424]: monmap e1: 1 mons at {smithi059=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-23T08:43:10.313 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:10 smithi059 ceph-mon[28424]: fsmap 2024-01-23T08:43:10.313 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:10 smithi059 ceph-mon[28424]: osdmap e1: 0 total, 0 up, 0 in 2024-01-23T08:43:10.313 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:10 smithi059 ceph-mon[28424]: mgrmap e1: no daemons active 2024-01-23T08:43:11.025 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.414+0000 7f812d775700 1 Processor -- start 2024-01-23T08:43:11.025 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.414+0000 7f812d775700 1 -- start start 2024-01-23T08:43:11.025 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.414+0000 7f812d775700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8128101410 0x7f8128101820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:11.025 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.414+0000 7f812d775700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8128101df0 con 0x7f8128101410 2024-01-23T08:43:11.025 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.414+0000 7f8126ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8128101410 0x7f8128101820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:11.025 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.414+0000 7f8126ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8128101410 0x7f8128101820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60042/0 (socket says 172.21.15.59:60042) 2024-01-23T08:43:11.025 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.414+0000 7f8126ffd700 1 -- 172.21.15.59:0/36540670 learned_addr learned my addr 172.21.15.59:0/36540670 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:11.025 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.415+0000 7f8126ffd700 1 -- 172.21.15.59:0/36540670 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8128102620 con 0x7f8128101410 2024-01-23T08:43:11.025 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.415+0000 7f8126ffd700 1 --2- 172.21.15.59:0/36540670 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8128101410 0x7f8128101820 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f811800bb80 tx=0x7f811800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f3d8e0a477b32b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:11.025 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.415+0000 7f8125ffb700 1 -- 172.21.15.59:0/36540670 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f811800dd00 con 0x7f8128101410 2024-01-23T08:43:11.026 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.415+0000 7f8125ffb700 1 -- 172.21.15.59:0/36540670 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f811800de60 con 0x7f8128101410 2024-01-23T08:43:11.026 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.415+0000 7f8125ffb700 1 -- 172.21.15.59:0/36540670 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8118018840 con 0x7f8128101410 2024-01-23T08:43:11.026 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.416+0000 7f812d775700 1 -- 172.21.15.59:0/36540670 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8128101410 msgr2=0x7f8128101820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:11.026 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.416+0000 7f812d775700 1 --2- 172.21.15.59:0/36540670 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8128101410 0x7f8128101820 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f811800bb80 tx=0x7f811800b4f0 comp rx=0 tx=0).stop 2024-01-23T08:43:11.026 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.416+0000 7f812d775700 1 -- 172.21.15.59:0/36540670 shutdown_connections 2024-01-23T08:43:11.026 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.416+0000 7f812d775700 1 --2- 172.21.15.59:0/36540670 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8128101410 0x7f8128101820 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:11.026 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.416+0000 7f812d775700 1 -- 172.21.15.59:0/36540670 >> 172.21.15.59:0/36540670 conn(0x7f81280fcad0 msgr2=0x7f81280fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:11.026 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.416+0000 7f812d775700 1 -- 172.21.15.59:0/36540670 shutdown_connections 2024-01-23T08:43:11.026 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.416+0000 7f812d775700 1 -- 172.21.15.59:0/36540670 wait complete. 2024-01-23T08:43:11.026 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.416+0000 7f812d775700 1 Processor -- start 2024-01-23T08:43:11.026 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.416+0000 7f812d775700 1 -- start start 2024-01-23T08:43:11.026 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.416+0000 7f812d775700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8128101410 0x7f81281133e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:11.026 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.416+0000 7f812d775700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8128113920 con 0x7f8128101410 2024-01-23T08:43:11.027 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.417+0000 7f8126ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8128101410 0x7f81281133e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:11.027 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.417+0000 7f8126ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8128101410 0x7f81281133e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60048/0 (socket says 172.21.15.59:60048) 2024-01-23T08:43:11.027 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.417+0000 7f8126ffd700 1 -- 172.21.15.59:0/2517965731 learned_addr learned my addr 172.21.15.59:0/2517965731 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:11.027 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.417+0000 7f8126ffd700 1 -- 172.21.15.59:0/2517965731 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811800b790 con 0x7f8128101410 2024-01-23T08:43:11.027 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.417+0000 7f8126ffd700 1 --2- 172.21.15.59:0/2517965731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8128101410 0x7f81281133e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8118006b00 tx=0x7f811800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:11.027 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.417+0000 7f810f7fe700 1 -- 172.21.15.59:0/2517965731 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8118010800 con 0x7f8128101410 2024-01-23T08:43:11.027 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.417+0000 7f810f7fe700 1 -- 172.21.15.59:0/2517965731 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f81180223f0 con 0x7f8128101410 2024-01-23T08:43:11.027 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.417+0000 7f812d775700 1 -- 172.21.15.59:0/2517965731 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8128113b20 con 0x7f8128101410 2024-01-23T08:43:11.027 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.417+0000 7f810f7fe700 1 -- 172.21.15.59:0/2517965731 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8118021550 con 0x7f8128101410 2024-01-23T08:43:11.027 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.417+0000 7f812d775700 1 -- 172.21.15.59:0/2517965731 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8128113f30 con 0x7f8128101410 2024-01-23T08:43:11.027 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.417+0000 7f810f7fe700 1 -- 172.21.15.59:0/2517965731 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f8118021810 con 0x7f8128101410 2024-01-23T08:43:11.027 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.417+0000 7f810f7fe700 1 -- 172.21.15.59:0/2517965731 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f8118014070 con 0x7f8128101410 2024-01-23T08:43:11.027 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.418+0000 7f812d775700 1 -- 172.21.15.59:0/2517965731 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8128108f60 con 0x7f8128101410 2024-01-23T08:43:11.027 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.420+0000 7f810f7fe700 1 -- 172.21.15.59:0/2517965731 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f8118022960 con 0x7f8128101410 2024-01-23T08:43:11.027 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.489+0000 7f812d775700 1 -- 172.21.15.59:0/2517965731 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f81281032c0 con 0x7f8128101410 2024-01-23T08:43:11.028 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.494+0000 7f810f7fe700 1 -- 172.21.15.59:0/2517965731 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f8118021c50 con 0x7f8128101410 2024-01-23T08:43:11.028 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.495+0000 7f812d775700 1 -- 172.21.15.59:0/2517965731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8128101410 msgr2=0x7f81281133e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:11.028 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.495+0000 7f812d775700 1 --2- 172.21.15.59:0/2517965731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8128101410 0x7f81281133e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8118006b00 tx=0x7f811800b4f0 comp rx=0 tx=0).stop 2024-01-23T08:43:11.028 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.495+0000 7f812d775700 1 -- 172.21.15.59:0/2517965731 shutdown_connections 2024-01-23T08:43:11.028 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.495+0000 7f812d775700 1 --2- 172.21.15.59:0/2517965731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8128101410 0x7f81281133e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:11.028 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.495+0000 7f812d775700 1 -- 172.21.15.59:0/2517965731 >> 172.21.15.59:0/2517965731 conn(0x7f81280fcad0 msgr2=0x7f81281060e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:11.028 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.495+0000 7f812d775700 1 -- 172.21.15.59:0/2517965731 shutdown_connections 2024-01-23T08:43:11.028 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:10.495+0000 7f812d775700 1 -- 172.21.15.59:0/2517965731 wait complete. 2024-01-23T08:43:11.028 INFO:teuthology.orchestra.run.smithi059.stdout:Wrote config to /etc/ceph/ceph.conf 2024-01-23T08:43:11.028 INFO:teuthology.orchestra.run.smithi059.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-01-23T08:43:11.028 INFO:teuthology.orchestra.run.smithi059.stdout:Creating mgr... 2024-01-23T08:43:11.028 INFO:teuthology.orchestra.run.smithi059.stdout:Verifying port 9283 ... 2024-01-23T08:43:11.191 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@mgr.smithi059.bzqirf 2024-01-23T08:43:11.192 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@mgr.smithi059.bzqirf.service: Unit ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@mgr.smithi059.bzqirf.service not loaded. 2024-01-23T08:43:11.347 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715.target.wants/ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@mgr.smithi059.bzqirf.service → /etc/systemd/system/ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@.service. 2024-01-23T08:43:11.669 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:11 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/2517965731' entity='client.admin' 2024-01-23T08:43:11.869 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-01-23T08:43:11.869 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout disabled 2024-01-23T08:43:11.877 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-01-23T08:43:11.877 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout inactive 2024-01-23T08:43:11.877 INFO:teuthology.orchestra.run.smithi059.stdout:firewalld.service is not enabled 2024-01-23T08:43:11.877 INFO:teuthology.orchestra.run.smithi059.stdout:Not possible to enable service . firewalld.service is not available 2024-01-23T08:43:11.886 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-01-23T08:43:11.887 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout disabled 2024-01-23T08:43:11.895 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-01-23T08:43:11.895 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout inactive 2024-01-23T08:43:11.895 INFO:teuthology.orchestra.run.smithi059.stdout:firewalld.service is not enabled 2024-01-23T08:43:11.895 INFO:teuthology.orchestra.run.smithi059.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-01-23T08:43:11.895 INFO:teuthology.orchestra.run.smithi059.stdout:Waiting for mgr to start... 2024-01-23T08:43:11.895 INFO:teuthology.orchestra.run.smithi059.stdout:Waiting for mgr... 2024-01-23T08:43:12.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:12 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/145983836' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-23T08:43:13.035 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-01-23T08:43:13.035 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout { 2024-01-23T08:43:13.035 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "fsid": "3fc07a38-b9cb-11ee-95b1-87774f69a715", 2024-01-23T08:43:13.035 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "health": { 2024-01-23T08:43:13.035 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-01-23T08:43:13.035 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-01-23T08:43:13.035 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-01-23T08:43:13.035 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:13.035 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-01-23T08:43:13.036 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-01-23T08:43:13.036 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 0 2024-01-23T08:43:13.036 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2024-01-23T08:43:13.036 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-01-23T08:43:13.036 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "smithi059" 2024-01-23T08:43:13.036 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2024-01-23T08:43:13.036 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-01-23T08:43:13.036 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "monmap": { 2024-01-23T08:43:13.036 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-23T08:43:13.036 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-01-23T08:43:13.036 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-01-23T08:43:13.036 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:13.036 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-01-23T08:43:13.036 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-23T08:43:13.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-01-23T08:43:13.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-01-23T08:43:13.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-01-23T08:43:13.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-01-23T08:43:13.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-01-23T08:43:13.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-01-23T08:43:13.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:13.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-01-23T08:43:13.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-01-23T08:43:13.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-01-23T08:43:13.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-01-23T08:43:13.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-01-23T08:43:13.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-01-23T08:43:13.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-01-23T08:43:13.037 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-01-23T08:43:13.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-01-23T08:43:13.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:13.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-01-23T08:43:13.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-23T08:43:13.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-01-23T08:43:13.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-01-23T08:43:13.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:13.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-01-23T08:43:13.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "available": false, 2024-01-23T08:43:13.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-01-23T08:43:13.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "modules": [ 2024-01-23T08:43:13.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "iostat", 2024-01-23T08:43:13.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "nfs", 2024-01-23T08:43:13.038 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "restful" 2024-01-23T08:43:13.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2024-01-23T08:43:13.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-23T08:43:13.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:13.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-01-23T08:43:13.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-23T08:43:13.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "modified": "2024-01-23T08:43:05.316609+0000", 2024-01-23T08:43:13.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-23T08:43:13.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:13.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-01-23T08:43:13.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout } 2024-01-23T08:43:13.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.366+0000 7fbf7d00c700 1 Processor -- start 2024-01-23T08:43:13.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.366+0000 7fbf7d00c700 1 -- start start 2024-01-23T08:43:13.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.366+0000 7fbf7d00c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf78101400 0x7fbf78101810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:13.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.366+0000 7fbf7d00c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf78101de0 con 0x7fbf78101400 2024-01-23T08:43:13.039 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.366+0000 7fbf76d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf78101400 0x7fbf78101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:13.040 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.366+0000 7fbf76d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf78101400 0x7fbf78101810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60060/0 (socket says 172.21.15.59:60060) 2024-01-23T08:43:13.040 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.366+0000 7fbf76d9d700 1 -- 172.21.15.59:0/4281457617 learned_addr learned my addr 172.21.15.59:0/4281457617 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:13.040 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.367+0000 7fbf76d9d700 1 -- 172.21.15.59:0/4281457617 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf78102610 con 0x7fbf78101400 2024-01-23T08:43:13.040 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.367+0000 7fbf76d9d700 1 --2- 172.21.15.59:0/4281457617 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf78101400 0x7fbf78101810 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbf680098e0 tx=0x7fbf68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb67ece1fa458076 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:13.040 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.367+0000 7fbf75d9b700 1 -- 172.21.15.59:0/4281457617 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf6800b860 con 0x7fbf78101400 2024-01-23T08:43:13.040 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.367+0000 7fbf75d9b700 1 -- 172.21.15.59:0/4281457617 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fbf6800be30 con 0x7fbf78101400 2024-01-23T08:43:13.040 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.367+0000 7fbf75d9b700 1 -- 172.21.15.59:0/4281457617 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf68010380 con 0x7fbf78101400 2024-01-23T08:43:13.040 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.368+0000 7fbf7d00c700 1 -- 172.21.15.59:0/4281457617 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf78101400 msgr2=0x7fbf78101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:13.040 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.368+0000 7fbf7d00c700 1 --2- 172.21.15.59:0/4281457617 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf78101400 0x7fbf78101810 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbf680098e0 tx=0x7fbf68009250 comp rx=0 tx=0).stop 2024-01-23T08:43:13.040 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.368+0000 7fbf7d00c700 1 -- 172.21.15.59:0/4281457617 shutdown_connections 2024-01-23T08:43:13.040 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.368+0000 7fbf7d00c700 1 --2- 172.21.15.59:0/4281457617 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf78101400 0x7fbf78101810 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:13.040 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.368+0000 7fbf7d00c700 1 -- 172.21.15.59:0/4281457617 >> 172.21.15.59:0/4281457617 conn(0x7fbf780fcae0 msgr2=0x7fbf780feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:13.040 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.368+0000 7fbf7d00c700 1 -- 172.21.15.59:0/4281457617 shutdown_connections 2024-01-23T08:43:13.041 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.368+0000 7fbf7d00c700 1 -- 172.21.15.59:0/4281457617 wait complete. 2024-01-23T08:43:13.041 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.369+0000 7fbf7d00c700 1 Processor -- start 2024-01-23T08:43:13.041 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.369+0000 7fbf7d00c700 1 -- start start 2024-01-23T08:43:13.041 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.369+0000 7fbf7d00c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf78101400 0x7fbf78076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:13.041 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.369+0000 7fbf7d00c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf78113fd0 con 0x7fbf78101400 2024-01-23T08:43:13.041 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.369+0000 7fbf76d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf78101400 0x7fbf78076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:13.041 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.369+0000 7fbf76d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf78101400 0x7fbf78076a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60070/0 (socket says 172.21.15.59:60070) 2024-01-23T08:43:13.041 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.369+0000 7fbf76d9d700 1 -- 172.21.15.59:0/145983836 learned_addr learned my addr 172.21.15.59:0/145983836 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:13.041 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.369+0000 7fbf76d9d700 1 -- 172.21.15.59:0/145983836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf680094f0 con 0x7fbf78101400 2024-01-23T08:43:13.041 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.369+0000 7fbf76d9d700 1 --2- 172.21.15.59:0/145983836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf78101400 0x7fbf78076a50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbf680090b0 tx=0x7fbf68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:13.041 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.370+0000 7fbf5f7fe700 1 -- 172.21.15.59:0/145983836 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf68010a80 con 0x7fbf78101400 2024-01-23T08:43:13.041 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.370+0000 7fbf5f7fe700 1 -- 172.21.15.59:0/145983836 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fbf6801a430 con 0x7fbf78101400 2024-01-23T08:43:13.041 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.370+0000 7fbf7d00c700 1 -- 172.21.15.59:0/145983836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf78114110 con 0x7fbf78101400 2024-01-23T08:43:13.042 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.370+0000 7fbf5f7fe700 1 -- 172.21.15.59:0/145983836 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf680245b0 con 0x7fbf78101400 2024-01-23T08:43:13.042 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.370+0000 7fbf7d00c700 1 -- 172.21.15.59:0/145983836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf78114460 con 0x7fbf78101400 2024-01-23T08:43:13.042 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.370+0000 7fbf5f7fe700 1 -- 172.21.15.59:0/145983836 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fbf68024aa0 con 0x7fbf78101400 2024-01-23T08:43:13.042 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.370+0000 7fbf5f7fe700 1 -- 172.21.15.59:0/145983836 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fbf6802daf0 con 0x7fbf78101400 2024-01-23T08:43:13.042 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.371+0000 7fbf7d00c700 1 -- 172.21.15.59:0/145983836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf7805f450 con 0x7fbf78101400 2024-01-23T08:43:13.042 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.373+0000 7fbf5f7fe700 1 -- 172.21.15.59:0/145983836 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fbf68029070 con 0x7fbf78101400 2024-01-23T08:43:13.042 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.446+0000 7fbf7d00c700 1 -- 172.21.15.59:0/145983836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbf7810fdb0 con 0x7fbf78101400 2024-01-23T08:43:13.042 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.446+0000 7fbf5f7fe700 1 -- 172.21.15.59:0/145983836 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fbf6801a960 con 0x7fbf78101400 2024-01-23T08:43:13.042 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.447+0000 7fbf7d00c700 1 -- 172.21.15.59:0/145983836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf78101400 msgr2=0x7fbf78076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:13.042 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.447+0000 7fbf7d00c700 1 --2- 172.21.15.59:0/145983836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf78101400 0x7fbf78076a50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbf680090b0 tx=0x7fbf68009250 comp rx=0 tx=0).stop 2024-01-23T08:43:13.042 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.447+0000 7fbf7d00c700 1 -- 172.21.15.59:0/145983836 shutdown_connections 2024-01-23T08:43:13.042 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.447+0000 7fbf7d00c700 1 --2- 172.21.15.59:0/145983836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf78101400 0x7fbf78076a50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:13.042 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.447+0000 7fbf7d00c700 1 -- 172.21.15.59:0/145983836 >> 172.21.15.59:0/145983836 conn(0x7fbf780fcae0 msgr2=0x7fbf780fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:13.042 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.447+0000 7fbf7d00c700 1 -- 172.21.15.59:0/145983836 shutdown_connections 2024-01-23T08:43:13.042 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:12.447+0000 7fbf7d00c700 1 -- 172.21.15.59:0/145983836 wait complete. 2024-01-23T08:43:13.043 INFO:teuthology.orchestra.run.smithi059.stdout:mgr not available, waiting (1/15)... 2024-01-23T08:43:16.121 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:15 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/446752956' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-23T08:43:16.139 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-01-23T08:43:16.140 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout { 2024-01-23T08:43:16.140 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "fsid": "3fc07a38-b9cb-11ee-95b1-87774f69a715", 2024-01-23T08:43:16.140 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "health": { 2024-01-23T08:43:16.140 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-01-23T08:43:16.140 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-01-23T08:43:16.140 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-01-23T08:43:16.140 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:16.140 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-01-23T08:43:16.140 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-01-23T08:43:16.140 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 0 2024-01-23T08:43:16.140 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2024-01-23T08:43:16.140 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-01-23T08:43:16.140 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "smithi059" 2024-01-23T08:43:16.140 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2024-01-23T08:43:16.140 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-01-23T08:43:16.141 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "monmap": { 2024-01-23T08:43:16.141 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-23T08:43:16.141 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-01-23T08:43:16.141 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-01-23T08:43:16.141 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:16.141 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-01-23T08:43:16.141 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-23T08:43:16.141 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-01-23T08:43:16.141 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-01-23T08:43:16.141 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-01-23T08:43:16.141 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-01-23T08:43:16.141 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-01-23T08:43:16.141 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-01-23T08:43:16.141 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:16.142 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-01-23T08:43:16.142 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-01-23T08:43:16.142 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-01-23T08:43:16.142 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-01-23T08:43:16.142 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-01-23T08:43:16.142 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-01-23T08:43:16.142 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-01-23T08:43:16.142 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-01-23T08:43:16.142 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-01-23T08:43:16.142 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:16.142 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-01-23T08:43:16.142 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-23T08:43:16.142 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-01-23T08:43:16.142 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-01-23T08:43:16.142 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:16.142 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-01-23T08:43:16.142 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "available": false, 2024-01-23T08:43:16.143 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-01-23T08:43:16.143 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "modules": [ 2024-01-23T08:43:16.143 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "iostat", 2024-01-23T08:43:16.143 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "nfs", 2024-01-23T08:43:16.143 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "restful" 2024-01-23T08:43:16.143 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2024-01-23T08:43:16.143 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-23T08:43:16.143 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:16.143 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-01-23T08:43:16.143 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-23T08:43:16.143 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "modified": "2024-01-23T08:43:05.316609+0000", 2024-01-23T08:43:16.143 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-23T08:43:16.143 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:16.143 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-01-23T08:43:16.143 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout } 2024-01-23T08:43:16.143 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.504+0000 7fe3e289a700 1 Processor -- start 2024-01-23T08:43:16.144 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.504+0000 7fe3e289a700 1 -- start start 2024-01-23T08:43:16.144 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.504+0000 7fe3e289a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3dc101400 0x7fe3dc101810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:16.144 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.504+0000 7fe3e289a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc101de0 con 0x7fe3dc101400 2024-01-23T08:43:16.144 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.504+0000 7fe3dbfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3dc101400 0x7fe3dc101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:16.144 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.504+0000 7fe3dbfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3dc101400 0x7fe3dc101810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60072/0 (socket says 172.21.15.59:60072) 2024-01-23T08:43:16.144 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.504+0000 7fe3dbfff700 1 -- 172.21.15.59:0/1366647492 learned_addr learned my addr 172.21.15.59:0/1366647492 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:16.144 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.504+0000 7fe3dbfff700 1 -- 172.21.15.59:0/1366647492 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3dc102610 con 0x7fe3dc101400 2024-01-23T08:43:16.144 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.505+0000 7fe3dbfff700 1 --2- 172.21.15.59:0/1366647492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3dc101400 0x7fe3dc101810 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc00bb80 tx=0x7fe3cc00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=928047f9fdec1e15 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:16.144 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.505+0000 7fe3daffd700 1 -- 172.21.15.59:0/1366647492 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3cc00dd00 con 0x7fe3dc101400 2024-01-23T08:43:16.144 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.505+0000 7fe3daffd700 1 -- 172.21.15.59:0/1366647492 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe3cc00de60 con 0x7fe3dc101400 2024-01-23T08:43:16.144 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.505+0000 7fe3daffd700 1 -- 172.21.15.59:0/1366647492 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3cc018840 con 0x7fe3dc101400 2024-01-23T08:43:16.144 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.505+0000 7fe3e289a700 1 -- 172.21.15.59:0/1366647492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3dc101400 msgr2=0x7fe3dc101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:16.144 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.505+0000 7fe3e289a700 1 --2- 172.21.15.59:0/1366647492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3dc101400 0x7fe3dc101810 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc00bb80 tx=0x7fe3cc00b4f0 comp rx=0 tx=0).stop 2024-01-23T08:43:16.145 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.505+0000 7fe3e289a700 1 -- 172.21.15.59:0/1366647492 shutdown_connections 2024-01-23T08:43:16.145 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.505+0000 7fe3e289a700 1 --2- 172.21.15.59:0/1366647492 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3dc101400 0x7fe3dc101810 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:16.145 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.505+0000 7fe3e289a700 1 -- 172.21.15.59:0/1366647492 >> 172.21.15.59:0/1366647492 conn(0x7fe3dc0fcae0 msgr2=0x7fe3dc0feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:16.145 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.506+0000 7fe3e289a700 1 -- 172.21.15.59:0/1366647492 shutdown_connections 2024-01-23T08:43:16.145 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.506+0000 7fe3e289a700 1 -- 172.21.15.59:0/1366647492 wait complete. 2024-01-23T08:43:16.145 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.506+0000 7fe3e289a700 1 Processor -- start 2024-01-23T08:43:16.145 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.506+0000 7fe3e289a700 1 -- start start 2024-01-23T08:43:16.145 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.506+0000 7fe3e289a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3dc101400 0x7fe3dc10ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:16.145 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.506+0000 7fe3e289a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc10f470 con 0x7fe3dc101400 2024-01-23T08:43:16.145 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.506+0000 7fe3dbfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3dc101400 0x7fe3dc10ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:16.145 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.506+0000 7fe3dbfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3dc101400 0x7fe3dc10ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60080/0 (socket says 172.21.15.59:60080) 2024-01-23T08:43:16.145 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.506+0000 7fe3dbfff700 1 -- 172.21.15.59:0/446752956 learned_addr learned my addr 172.21.15.59:0/446752956 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:16.145 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.507+0000 7fe3dbfff700 1 -- 172.21.15.59:0/446752956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3cc00b790 con 0x7fe3dc101400 2024-01-23T08:43:16.145 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.507+0000 7fe3dbfff700 1 --2- 172.21.15.59:0/446752956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3dc101400 0x7fe3dc10ef30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc006b00 tx=0x7fe3cc00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:16.146 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.507+0000 7fe3d8ff9700 1 -- 172.21.15.59:0/446752956 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3cc010a50 con 0x7fe3dc101400 2024-01-23T08:43:16.146 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.507+0000 7fe3d8ff9700 1 -- 172.21.15.59:0/446752956 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe3cc01d050 con 0x7fe3dc101400 2024-01-23T08:43:16.146 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.507+0000 7fe3e289a700 1 -- 172.21.15.59:0/446752956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3dc10f670 con 0x7fe3dc101400 2024-01-23T08:43:16.146 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.507+0000 7fe3d8ff9700 1 -- 172.21.15.59:0/446752956 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3cc022590 con 0x7fe3dc101400 2024-01-23T08:43:16.146 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.507+0000 7fe3e289a700 1 -- 172.21.15.59:0/446752956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3dc10fa80 con 0x7fe3dc101400 2024-01-23T08:43:16.146 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.507+0000 7fe3d8ff9700 1 -- 172.21.15.59:0/446752956 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fe3cc022850 con 0x7fe3dc101400 2024-01-23T08:43:16.146 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.507+0000 7fe3d8ff9700 1 -- 172.21.15.59:0/446752956 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe3cc014070 con 0x7fe3dc101400 2024-01-23T08:43:16.146 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.508+0000 7fe3e289a700 1 -- 172.21.15.59:0/446752956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3dc108f50 con 0x7fe3dc101400 2024-01-23T08:43:16.146 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.512+0000 7fe3d8ff9700 1 -- 172.21.15.59:0/446752956 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fe3cc010bb0 con 0x7fe3dc101400 2024-01-23T08:43:16.146 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.597+0000 7fe3e289a700 1 -- 172.21.15.59:0/446752956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe3dc1032b0 con 0x7fe3dc101400 2024-01-23T08:43:16.146 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.597+0000 7fe3d8ff9700 1 -- 172.21.15.59:0/446752956 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fe3cc022c90 con 0x7fe3dc101400 2024-01-23T08:43:16.146 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.598+0000 7fe3e289a700 1 -- 172.21.15.59:0/446752956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3dc101400 msgr2=0x7fe3dc10ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:16.146 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.598+0000 7fe3e289a700 1 --2- 172.21.15.59:0/446752956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3dc101400 0x7fe3dc10ef30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc006b00 tx=0x7fe3cc00b4f0 comp rx=0 tx=0).stop 2024-01-23T08:43:16.146 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.598+0000 7fe3e289a700 1 -- 172.21.15.59:0/446752956 shutdown_connections 2024-01-23T08:43:16.147 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.598+0000 7fe3e289a700 1 --2- 172.21.15.59:0/446752956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3dc101400 0x7fe3dc10ef30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:16.147 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.599+0000 7fe3e289a700 1 -- 172.21.15.59:0/446752956 >> 172.21.15.59:0/446752956 conn(0x7fe3dc0fcae0 msgr2=0x7fe3dc1060d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:16.147 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.599+0000 7fe3e289a700 1 -- 172.21.15.59:0/446752956 shutdown_connections 2024-01-23T08:43:16.147 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:15.599+0000 7fe3e289a700 1 -- 172.21.15.59:0/446752956 wait complete. 2024-01-23T08:43:16.147 INFO:teuthology.orchestra.run.smithi059.stdout:mgr not available, waiting (2/15)... 2024-01-23T08:43:18.957 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:18 smithi059 ceph-mon[28424]: Activating manager daemon smithi059.bzqirf 2024-01-23T08:43:18.957 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:18 smithi059 ceph-mon[28424]: mgrmap e2: smithi059.bzqirf(active, starting, since 0.00272427s) 2024-01-23T08:43:18.958 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:18 smithi059 ceph-mon[28424]: from='mgr.14100 172.21.15.59:0/844205584' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-23T08:43:18.958 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:18 smithi059 ceph-mon[28424]: from='mgr.14100 172.21.15.59:0/844205584' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-23T08:43:18.958 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:18 smithi059 ceph-mon[28424]: from='mgr.14100 172.21.15.59:0/844205584' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-23T08:43:18.958 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:18 smithi059 ceph-mon[28424]: from='mgr.14100 172.21.15.59:0/844205584' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-01-23T08:43:18.958 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:18 smithi059 ceph-mon[28424]: from='mgr.14100 172.21.15.59:0/844205584' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mgr metadata", "who": "smithi059.bzqirf", "id": "smithi059.bzqirf"}]: dispatch 2024-01-23T08:43:18.958 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:18 smithi059 ceph-mon[28424]: Manager daemon smithi059.bzqirf is now available 2024-01-23T08:43:18.958 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:18 smithi059 ceph-mon[28424]: from='mgr.14100 172.21.15.59:0/844205584' entity='mgr.smithi059.bzqirf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.bzqirf/mirror_snapshot_schedule"}]: dispatch 2024-01-23T08:43:18.958 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:18 smithi059 ceph-mon[28424]: from='mgr.14100 172.21.15.59:0/844205584' entity='mgr.smithi059.bzqirf' 2024-01-23T08:43:18.958 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:18 smithi059 ceph-mon[28424]: from='mgr.14100 172.21.15.59:0/844205584' entity='mgr.smithi059.bzqirf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.bzqirf/trash_purge_schedule"}]: dispatch 2024-01-23T08:43:18.958 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:18 smithi059 ceph-mon[28424]: from='mgr.14100 172.21.15.59:0/844205584' entity='mgr.smithi059.bzqirf' 2024-01-23T08:43:18.958 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:18 smithi059 ceph-mon[28424]: from='mgr.14100 172.21.15.59:0/844205584' entity='mgr.smithi059.bzqirf' 2024-01-23T08:43:19.380 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-01-23T08:43:19.380 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout { 2024-01-23T08:43:19.380 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "fsid": "3fc07a38-b9cb-11ee-95b1-87774f69a715", 2024-01-23T08:43:19.380 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "health": { 2024-01-23T08:43:19.380 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-01-23T08:43:19.380 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-01-23T08:43:19.380 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-01-23T08:43:19.380 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:19.380 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-01-23T08:43:19.380 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-01-23T08:43:19.380 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 0 2024-01-23T08:43:19.381 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2024-01-23T08:43:19.381 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-01-23T08:43:19.381 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "smithi059" 2024-01-23T08:43:19.381 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2024-01-23T08:43:19.381 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-01-23T08:43:19.381 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "monmap": { 2024-01-23T08:43:19.381 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-23T08:43:19.381 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-01-23T08:43:19.381 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-01-23T08:43:19.381 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:19.381 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-01-23T08:43:19.381 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-23T08:43:19.381 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-01-23T08:43:19.381 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-01-23T08:43:19.381 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-01-23T08:43:19.390 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-01-23T08:43:19.390 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-01-23T08:43:19.390 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-01-23T08:43:19.390 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:19.390 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-01-23T08:43:19.390 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-01-23T08:43:19.390 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-01-23T08:43:19.390 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-01-23T08:43:19.390 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-01-23T08:43:19.390 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-01-23T08:43:19.390 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-01-23T08:43:19.390 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-01-23T08:43:19.390 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-01-23T08:43:19.390 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:19.390 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-01-23T08:43:19.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-23T08:43:19.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-01-23T08:43:19.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-01-23T08:43:19.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:19.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-01-23T08:43:19.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "available": false, 2024-01-23T08:43:19.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-01-23T08:43:19.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "modules": [ 2024-01-23T08:43:19.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "iostat", 2024-01-23T08:43:19.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "nfs", 2024-01-23T08:43:19.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "restful" 2024-01-23T08:43:19.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2024-01-23T08:43:19.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-23T08:43:19.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:19.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-01-23T08:43:19.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-23T08:43:19.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "modified": "2024-01-23T08:43:05.316609+0000", 2024-01-23T08:43:19.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-23T08:43:19.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:19.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-01-23T08:43:19.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout } 2024-01-23T08:43:19.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.595+0000 7f388caef700 1 Processor -- start 2024-01-23T08:43:19.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.595+0000 7f388caef700 1 -- start start 2024-01-23T08:43:19.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.595+0000 7f388caef700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3888102500 0x7f3888102910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:19.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.595+0000 7f388caef700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3888102e50 con 0x7f3888102500 2024-01-23T08:43:19.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.595+0000 7f388659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3888102500 0x7f3888102910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:19.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.595+0000 7f388659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3888102500 0x7f3888102910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56226/0 (socket says 172.21.15.59:56226) 2024-01-23T08:43:19.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.595+0000 7f388659c700 1 -- 172.21.15.59:0/1326310304 learned_addr learned my addr 172.21.15.59:0/1326310304 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:19.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.596+0000 7f388659c700 1 -- 172.21.15.59:0/1326310304 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3888102f90 con 0x7f3888102500 2024-01-23T08:43:19.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.596+0000 7f388659c700 1 --2- 172.21.15.59:0/1326310304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3888102500 0x7f3888102910 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f38780098e0 tx=0x7f3878009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58266f5d261a6a8c server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:19.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.596+0000 7f388559a700 1 -- 172.21.15.59:0/1326310304 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f387800b860 con 0x7f3888102500 2024-01-23T08:43:19.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.596+0000 7f388559a700 1 -- 172.21.15.59:0/1326310304 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f387800be30 con 0x7f3888102500 2024-01-23T08:43:19.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.596+0000 7f388559a700 1 -- 172.21.15.59:0/1326310304 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3878010380 con 0x7f3888102500 2024-01-23T08:43:19.393 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.596+0000 7f388caef700 1 -- 172.21.15.59:0/1326310304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3888102500 msgr2=0x7f3888102910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:19.393 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.596+0000 7f388caef700 1 --2- 172.21.15.59:0/1326310304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3888102500 0x7f3888102910 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f38780098e0 tx=0x7f3878009250 comp rx=0 tx=0).stop 2024-01-23T08:43:19.393 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.596+0000 7f388caef700 1 -- 172.21.15.59:0/1326310304 shutdown_connections 2024-01-23T08:43:19.393 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.596+0000 7f388caef700 1 --2- 172.21.15.59:0/1326310304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3888102500 0x7f3888102910 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:19.393 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.596+0000 7f388caef700 1 -- 172.21.15.59:0/1326310304 >> 172.21.15.59:0/1326310304 conn(0x7f38880fdca0 msgr2=0x7f38881000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:19.393 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.596+0000 7f388caef700 1 -- 172.21.15.59:0/1326310304 shutdown_connections 2024-01-23T08:43:19.393 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.596+0000 7f388caef700 1 -- 172.21.15.59:0/1326310304 wait complete. 2024-01-23T08:43:19.393 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.597+0000 7f388caef700 1 Processor -- start 2024-01-23T08:43:19.393 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.597+0000 7f388caef700 1 -- start start 2024-01-23T08:43:19.393 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.597+0000 7f388caef700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3888102500 0x7f3888078b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:19.393 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.597+0000 7f388caef700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3888079090 con 0x7f3888102500 2024-01-23T08:43:19.393 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.597+0000 7f388659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3888102500 0x7f3888078b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:19.393 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.597+0000 7f388659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3888102500 0x7f3888078b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56230/0 (socket says 172.21.15.59:56230) 2024-01-23T08:43:19.393 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.597+0000 7f388659c700 1 -- 172.21.15.59:0/1898383441 learned_addr learned my addr 172.21.15.59:0/1898383441 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:19.393 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.597+0000 7f388659c700 1 -- 172.21.15.59:0/1898383441 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38780094f0 con 0x7f3888102500 2024-01-23T08:43:19.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.598+0000 7f388659c700 1 --2- 172.21.15.59:0/1898383441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3888102500 0x7f3888078b50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3878006b00 tx=0x7f3878009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:19.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.598+0000 7f3876ffd700 1 -- 172.21.15.59:0/1898383441 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38780109b0 con 0x7f3888102500 2024-01-23T08:43:19.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.598+0000 7f3876ffd700 1 -- 172.21.15.59:0/1898383441 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f387801a430 con 0x7f3888102500 2024-01-23T08:43:19.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.598+0000 7f388caef700 1 -- 172.21.15.59:0/1898383441 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38880771b0 con 0x7f3888102500 2024-01-23T08:43:19.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.598+0000 7f3876ffd700 1 -- 172.21.15.59:0/1898383441 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38780245b0 con 0x7f3888102500 2024-01-23T08:43:19.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.598+0000 7f388caef700 1 -- 172.21.15.59:0/1898383441 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3888077620 con 0x7f3888102500 2024-01-23T08:43:19.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.598+0000 7f3876ffd700 1 -- 172.21.15.59:0/1898383441 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 2) v1 ==== 43037+0+0 (secure 0 0 0) 0x7f38780247e0 con 0x7f3888102500 2024-01-23T08:43:19.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.599+0000 7f3876ffd700 1 -- 172.21.15.59:0/1898383441 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f387804a7a0 con 0x7f3888102500 2024-01-23T08:43:19.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.599+0000 7f388caef700 1 -- 172.21.15.59:0/1898383441 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f388805f450 con 0x7f3888102500 2024-01-23T08:43:19.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.601+0000 7f3876ffd700 1 -- 172.21.15.59:0/1898383441 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f3878024a90 con 0x7f3888102500 2024-01-23T08:43:19.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.672+0000 7f388caef700 1 -- 172.21.15.59:0/1898383441 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f388810ada0 con 0x7f3888102500 2024-01-23T08:43:19.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.673+0000 7f3876ffd700 1 -- 172.21.15.59:0/1898383441 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f387802cb80 con 0x7f3888102500 2024-01-23T08:43:19.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.673+0000 7f388caef700 1 -- 172.21.15.59:0/1898383441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3888102500 msgr2=0x7f3888078b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:19.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.673+0000 7f388caef700 1 --2- 172.21.15.59:0/1898383441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3888102500 0x7f3888078b50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3878006b00 tx=0x7f3878009250 comp rx=0 tx=0).stop 2024-01-23T08:43:19.395 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.673+0000 7f388caef700 1 -- 172.21.15.59:0/1898383441 shutdown_connections 2024-01-23T08:43:19.395 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.673+0000 7f388caef700 1 --2- 172.21.15.59:0/1898383441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3888102500 0x7f3888078b50 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:19.395 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.673+0000 7f388caef700 1 -- 172.21.15.59:0/1898383441 >> 172.21.15.59:0/1898383441 conn(0x7f38880fdca0 msgr2=0x7f38880feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:19.395 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.673+0000 7f388caef700 1 -- 172.21.15.59:0/1898383441 shutdown_connections 2024-01-23T08:43:19.395 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:18.673+0000 7f388caef700 1 -- 172.21.15.59:0/1898383441 wait complete. 2024-01-23T08:43:19.395 INFO:teuthology.orchestra.run.smithi059.stdout:mgr not available, waiting (3/15)... 2024-01-23T08:43:19.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:19 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/1898383441' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-23T08:43:19.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:19 smithi059 ceph-mon[28424]: mgrmap e3: smithi059.bzqirf(active, since 1.00712s) 2024-01-23T08:43:22.068 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:21 smithi059 ceph-mon[28424]: mgrmap e4: smithi059.bzqirf(active, since 2s) 2024-01-23T08:43:22.817 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-01-23T08:43:22.820 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout { 2024-01-23T08:43:22.821 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "fsid": "3fc07a38-b9cb-11ee-95b1-87774f69a715", 2024-01-23T08:43:22.821 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "health": { 2024-01-23T08:43:22.821 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-01-23T08:43:22.821 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-01-23T08:43:22.821 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-01-23T08:43:22.821 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:22.821 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-01-23T08:43:22.821 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-01-23T08:43:22.821 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 0 2024-01-23T08:43:22.821 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2024-01-23T08:43:22.821 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-01-23T08:43:22.821 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "smithi059" 2024-01-23T08:43:22.821 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2024-01-23T08:43:22.821 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-01-23T08:43:22.822 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "monmap": { 2024-01-23T08:43:22.822 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-23T08:43:22.822 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-01-23T08:43:22.822 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-01-23T08:43:22.822 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:22.822 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-01-23T08:43:22.822 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-23T08:43:22.822 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-01-23T08:43:22.822 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-01-23T08:43:22.822 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-01-23T08:43:22.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-01-23T08:43:22.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-01-23T08:43:22.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-01-23T08:43:22.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:22.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-01-23T08:43:22.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-01-23T08:43:22.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-01-23T08:43:22.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-01-23T08:43:22.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-01-23T08:43:22.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-01-23T08:43:22.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-01-23T08:43:22.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-01-23T08:43:22.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-01-23T08:43:22.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:22.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-01-23T08:43:22.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-23T08:43:22.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-01-23T08:43:22.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-01-23T08:43:22.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:22.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-01-23T08:43:22.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "available": true, 2024-01-23T08:43:22.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-01-23T08:43:22.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "modules": [ 2024-01-23T08:43:22.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "iostat", 2024-01-23T08:43:22.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "nfs", 2024-01-23T08:43:22.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "restful" 2024-01-23T08:43:22.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2024-01-23T08:43:22.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-23T08:43:22.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:22.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-01-23T08:43:22.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-23T08:43:22.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "modified": "2024-01-23T08:43:05.316609+0000", 2024-01-23T08:43:22.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-23T08:43:22.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2024-01-23T08:43:22.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-01-23T08:43:22.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout } 2024-01-23T08:43:22.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.793+0000 7f35d802f700 1 Processor -- start 2024-01-23T08:43:22.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.793+0000 7f35d802f700 1 -- start start 2024-01-23T08:43:22.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.793+0000 7f35d802f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35d0102500 0x7f35d0102910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:22.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.793+0000 7f35d802f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d0102e50 con 0x7f35d0102500 2024-01-23T08:43:22.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.793+0000 7f35d5dcb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35d0102500 0x7f35d0102910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:22.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.793+0000 7f35d5dcb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35d0102500 0x7f35d0102910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56244/0 (socket says 172.21.15.59:56244) 2024-01-23T08:43:22.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.793+0000 7f35d5dcb700 1 -- 172.21.15.59:0/2784585822 learned_addr learned my addr 172.21.15.59:0/2784585822 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:22.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.794+0000 7f35d5dcb700 1 -- 172.21.15.59:0/2784585822 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d0102f90 con 0x7f35d0102500 2024-01-23T08:43:22.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.794+0000 7f35d5dcb700 1 --2- 172.21.15.59:0/2784585822 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35d0102500 0x7f35d0102910 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f35cc0098e0 tx=0x7f35cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c63e493dad7f001d server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:22.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.794+0000 7f35d4dc9700 1 -- 172.21.15.59:0/2784585822 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35cc00b860 con 0x7f35d0102500 2024-01-23T08:43:22.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.794+0000 7f35d4dc9700 1 -- 172.21.15.59:0/2784585822 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f35cc00be30 con 0x7f35d0102500 2024-01-23T08:43:22.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.794+0000 7f35d4dc9700 1 -- 172.21.15.59:0/2784585822 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35cc010380 con 0x7f35d0102500 2024-01-23T08:43:22.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.795+0000 7f35d802f700 1 -- 172.21.15.59:0/2784585822 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35d0102500 msgr2=0x7f35d0102910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:22.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.795+0000 7f35d802f700 1 --2- 172.21.15.59:0/2784585822 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35d0102500 0x7f35d0102910 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f35cc0098e0 tx=0x7f35cc009250 comp rx=0 tx=0).stop 2024-01-23T08:43:22.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.795+0000 7f35d802f700 1 -- 172.21.15.59:0/2784585822 shutdown_connections 2024-01-23T08:43:22.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.795+0000 7f35d802f700 1 --2- 172.21.15.59:0/2784585822 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35d0102500 0x7f35d0102910 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:22.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.795+0000 7f35d802f700 1 -- 172.21.15.59:0/2784585822 >> 172.21.15.59:0/2784585822 conn(0x7f35d00fdca0 msgr2=0x7f35d01000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:22.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.795+0000 7f35d802f700 1 -- 172.21.15.59:0/2784585822 shutdown_connections 2024-01-23T08:43:22.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.795+0000 7f35d802f700 1 -- 172.21.15.59:0/2784585822 wait complete. 2024-01-23T08:43:22.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.795+0000 7f35d802f700 1 Processor -- start 2024-01-23T08:43:22.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.795+0000 7f35d802f700 1 -- start start 2024-01-23T08:43:22.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.795+0000 7f35d802f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35d0102500 0x7f35d0111490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:22.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.795+0000 7f35d802f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d01119d0 con 0x7f35d0102500 2024-01-23T08:43:22.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.796+0000 7f35d5dcb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35d0102500 0x7f35d0111490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:22.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.796+0000 7f35d5dcb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35d0102500 0x7f35d0111490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56258/0 (socket says 172.21.15.59:56258) 2024-01-23T08:43:22.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.796+0000 7f35d5dcb700 1 -- 172.21.15.59:0/1933536080 learned_addr learned my addr 172.21.15.59:0/1933536080 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:22.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.796+0000 7f35d5dcb700 1 -- 172.21.15.59:0/1933536080 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35cc0094f0 con 0x7f35d0102500 2024-01-23T08:43:22.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.796+0000 7f35d5dcb700 1 --2- 172.21.15.59:0/1933536080 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35d0102500 0x7f35d0111490 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f35cc000c00 tx=0x7f35cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:22.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.796+0000 7f35c67fc700 1 -- 172.21.15.59:0/1933536080 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35cc010a80 con 0x7f35d0102500 2024-01-23T08:43:22.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.796+0000 7f35c67fc700 1 -- 172.21.15.59:0/1933536080 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f35cc01a430 con 0x7f35d0102500 2024-01-23T08:43:22.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.796+0000 7f35d802f700 1 -- 172.21.15.59:0/1933536080 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35d0111bd0 con 0x7f35d0102500 2024-01-23T08:43:22.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.796+0000 7f35c67fc700 1 -- 172.21.15.59:0/1933536080 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35cc0245b0 con 0x7f35d0102500 2024-01-23T08:43:22.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.796+0000 7f35d802f700 1 -- 172.21.15.59:0/1933536080 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d0111fe0 con 0x7f35d0102500 2024-01-23T08:43:22.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.797+0000 7f35c67fc700 1 -- 172.21.15.59:0/1933536080 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 4) v1 ==== 43270+0+0 (secure 0 0 0) 0x7f35cc010be0 con 0x7f35d0102500 2024-01-23T08:43:22.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.797+0000 7f35d802f700 1 -- 172.21.15.59:0/1933536080 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35d010b380 con 0x7f35d0102500 2024-01-23T08:43:22.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.797+0000 7f35c67fc700 1 --2- 172.21.15.59:0/1933536080 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7f35bc0365a0 0x7f35bc038a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:22.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.798+0000 7f35c67fc700 1 -- 172.21.15.59:0/1933536080 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f35cc04a5b0 con 0x7f35d0102500 2024-01-23T08:43:22.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.798+0000 7f35d55ca700 1 --2- 172.21.15.59:0/1933536080 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7f35bc0365a0 0x7f35bc038a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:22.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.798+0000 7f35d55ca700 1 --2- 172.21.15.59:0/1933536080 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7f35bc0365a0 0x7f35bc038a50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f35c0009990 tx=0x7f35c0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:22.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:21.803+0000 7f35c67fc700 1 -- 172.21.15.59:0/1933536080 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f35cc018ca0 con 0x7f35d0102500 2024-01-23T08:43:22.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:22.070+0000 7f35d802f700 1 -- 172.21.15.59:0/1933536080 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f35d0064940 con 0x7f35d0102500 2024-01-23T08:43:22.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:22.070+0000 7f35c67fc700 1 -- 172.21.15.59:0/1933536080 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f35cc018ca0 con 0x7f35d0102500 2024-01-23T08:43:22.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:22.072+0000 7f35d802f700 1 -- 172.21.15.59:0/1933536080 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7f35bc0365a0 msgr2=0x7f35bc038a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:22.838 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:22.072+0000 7f35d802f700 1 --2- 172.21.15.59:0/1933536080 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7f35bc0365a0 0x7f35bc038a50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f35c0009990 tx=0x7f35c0006e40 comp rx=0 tx=0).stop 2024-01-23T08:43:22.838 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:22.072+0000 7f35d802f700 1 -- 172.21.15.59:0/1933536080 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35d0102500 msgr2=0x7f35d0111490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:22.838 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:22.072+0000 7f35d802f700 1 --2- 172.21.15.59:0/1933536080 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35d0102500 0x7f35d0111490 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f35cc000c00 tx=0x7f35cc009250 comp rx=0 tx=0).stop 2024-01-23T08:43:22.838 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:22.072+0000 7f35d802f700 1 -- 172.21.15.59:0/1933536080 shutdown_connections 2024-01-23T08:43:22.838 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:22.072+0000 7f35d802f700 1 --2- 172.21.15.59:0/1933536080 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7f35bc0365a0 0x7f35bc038a50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:22.838 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:22.072+0000 7f35d802f700 1 --2- 172.21.15.59:0/1933536080 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35d0102500 0x7f35d0111490 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:22.838 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:22.072+0000 7f35d802f700 1 -- 172.21.15.59:0/1933536080 >> 172.21.15.59:0/1933536080 conn(0x7f35d00fdca0 msgr2=0x7f35d00feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:22.838 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:22.072+0000 7f35d802f700 1 -- 172.21.15.59:0/1933536080 shutdown_connections 2024-01-23T08:43:22.838 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:22.072+0000 7f35d802f700 1 -- 172.21.15.59:0/1933536080 wait complete. 2024-01-23T08:43:22.838 INFO:teuthology.orchestra.run.smithi059.stdout:mgr is available 2024-01-23T08:43:23.117 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:22 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/1933536080' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-23T08:43:23.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:23 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/3423761289' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-23T08:43:23.997 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-01-23T08:43:23.997 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout [global] 2024-01-23T08:43:23.997 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout fsid = 3fc07a38-b9cb-11ee-95b1-87774f69a715 2024-01-23T08:43:23.998 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-01-23T08:43:23.998 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-01-23T08:43:23.998 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-01-23T08:43:23.998 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-01-23T08:43:23.998 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-01-23T08:43:23.998 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout [mgr] 2024-01-23T08:43:23.998 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-01-23T08:43:23.998 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2024-01-23T08:43:23.998 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout [osd] 2024-01-23T08:43:23.998 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-01-23T08:43:23.998 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-01-23T08:43:23.998 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-01-23T08:43:23.998 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-01-23T08:43:23.998 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-01-23T08:43:23.999 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.223+0000 7fed9379d700 1 Processor -- start 2024-01-23T08:43:23.999 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.223+0000 7fed9379d700 1 -- start start 2024-01-23T08:43:23.999 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.223+0000 7fed9379d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed8c102500 0x7fed8c102910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:23.999 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.223+0000 7fed9379d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c102e50 con 0x7fed8c102500 2024-01-23T08:43:23.999 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.224+0000 7fed91539700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed8c102500 0x7fed8c102910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:23.999 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.224+0000 7fed91539700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed8c102500 0x7fed8c102910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56266/0 (socket says 172.21.15.59:56266) 2024-01-23T08:43:23.999 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.224+0000 7fed91539700 1 -- 172.21.15.59:0/2861902327 learned_addr learned my addr 172.21.15.59:0/2861902327 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:23.999 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.224+0000 7fed91539700 1 -- 172.21.15.59:0/2861902327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed8c102f90 con 0x7fed8c102500 2024-01-23T08:43:23.999 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.224+0000 7fed91539700 1 --2- 172.21.15.59:0/2861902327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed8c102500 0x7fed8c102910 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fed800098e0 tx=0x7fed80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e70f20de88351af6 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:23.999 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.224+0000 7fed7ffff700 1 -- 172.21.15.59:0/2861902327 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed8000b860 con 0x7fed8c102500 2024-01-23T08:43:23.999 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.224+0000 7fed7ffff700 1 -- 172.21.15.59:0/2861902327 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fed8000be30 con 0x7fed8c102500 2024-01-23T08:43:23.999 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.224+0000 7fed7ffff700 1 -- 172.21.15.59:0/2861902327 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed80010380 con 0x7fed8c102500 2024-01-23T08:43:23.999 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.225+0000 7fed9379d700 1 -- 172.21.15.59:0/2861902327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed8c102500 msgr2=0x7fed8c102910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:23.999 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.225+0000 7fed9379d700 1 --2- 172.21.15.59:0/2861902327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed8c102500 0x7fed8c102910 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fed800098e0 tx=0x7fed80009250 comp rx=0 tx=0).stop 2024-01-23T08:43:24.000 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.225+0000 7fed9379d700 1 -- 172.21.15.59:0/2861902327 shutdown_connections 2024-01-23T08:43:24.000 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.225+0000 7fed9379d700 1 --2- 172.21.15.59:0/2861902327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed8c102500 0x7fed8c102910 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:24.000 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.225+0000 7fed9379d700 1 -- 172.21.15.59:0/2861902327 >> 172.21.15.59:0/2861902327 conn(0x7fed8c0fdc80 msgr2=0x7fed8c100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:24.000 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.225+0000 7fed9379d700 1 -- 172.21.15.59:0/2861902327 shutdown_connections 2024-01-23T08:43:24.000 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.225+0000 7fed9379d700 1 -- 172.21.15.59:0/2861902327 wait complete. 2024-01-23T08:43:24.000 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.225+0000 7fed9379d700 1 Processor -- start 2024-01-23T08:43:24.000 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.226+0000 7fed9379d700 1 -- start start 2024-01-23T08:43:24.000 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.226+0000 7fed9379d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed8c102500 0x7fed8c111460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:24.000 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.226+0000 7fed9379d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c1119a0 con 0x7fed8c102500 2024-01-23T08:43:24.000 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.226+0000 7fed91539700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed8c102500 0x7fed8c111460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:24.000 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.226+0000 7fed91539700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed8c102500 0x7fed8c111460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56282/0 (socket says 172.21.15.59:56282) 2024-01-23T08:43:24.000 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.226+0000 7fed91539700 1 -- 172.21.15.59:0/3423761289 learned_addr learned my addr 172.21.15.59:0/3423761289 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:24.000 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.226+0000 7fed91539700 1 -- 172.21.15.59:0/3423761289 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed800094f0 con 0x7fed8c102500 2024-01-23T08:43:24.001 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.226+0000 7fed91539700 1 --2- 172.21.15.59:0/3423761289 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed8c102500 0x7fed8c111460 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fed80000c00 tx=0x7fed80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:24.001 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.226+0000 7fed7dffb700 1 -- 172.21.15.59:0/3423761289 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed80010a80 con 0x7fed8c102500 2024-01-23T08:43:24.001 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.226+0000 7fed7dffb700 1 -- 172.21.15.59:0/3423761289 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fed8001a430 con 0x7fed8c102500 2024-01-23T08:43:24.001 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.226+0000 7fed9379d700 1 -- 172.21.15.59:0/3423761289 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed8c111ba0 con 0x7fed8c102500 2024-01-23T08:43:24.001 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.226+0000 7fed7dffb700 1 -- 172.21.15.59:0/3423761289 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed800245b0 con 0x7fed8c102500 2024-01-23T08:43:24.001 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.227+0000 7fed9379d700 1 -- 172.21.15.59:0/3423761289 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed8c111fb0 con 0x7fed8c102500 2024-01-23T08:43:24.001 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.227+0000 7fed7dffb700 1 -- 172.21.15.59:0/3423761289 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 4) v1 ==== 43270+0+0 (secure 0 0 0) 0x7fed80024870 con 0x7fed8c102500 2024-01-23T08:43:24.001 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.228+0000 7fed9379d700 1 -- 172.21.15.59:0/3423761289 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed8c10b380 con 0x7fed8c102500 2024-01-23T08:43:24.001 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.228+0000 7fed7dffb700 1 --2- 172.21.15.59:0/3423761289 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7fed780365a0 0x7fed78038a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:24.001 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.228+0000 7fed7dffb700 1 -- 172.21.15.59:0/3423761289 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fed8004a850 con 0x7fed8c102500 2024-01-23T08:43:24.001 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.228+0000 7fed90d38700 1 --2- 172.21.15.59:0/3423761289 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7fed780365a0 0x7fed78038a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:24.001 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.228+0000 7fed90d38700 1 --2- 172.21.15.59:0/3423761289 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7fed780365a0 0x7fed78038a50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fed88009990 tx=0x7fed88006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:24.001 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.233+0000 7fed7dffb700 1 -- 172.21.15.59:0/3423761289 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fed8002e960 con 0x7fed8c102500 2024-01-23T08:43:24.001 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.421+0000 7fed9379d700 1 -- 172.21.15.59:0/3423761289 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fed8c064940 con 0x7fed8c102500 2024-01-23T08:43:24.001 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.423+0000 7fed7dffb700 1 -- 172.21.15.59:0/3423761289 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fed80018b40 con 0x7fed8c102500 2024-01-23T08:43:24.002 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.425+0000 7fed9379d700 1 -- 172.21.15.59:0/3423761289 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7fed780365a0 msgr2=0x7fed78038a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:24.002 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.425+0000 7fed9379d700 1 --2- 172.21.15.59:0/3423761289 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7fed780365a0 0x7fed78038a50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fed88009990 tx=0x7fed88006e40 comp rx=0 tx=0).stop 2024-01-23T08:43:24.002 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.425+0000 7fed9379d700 1 -- 172.21.15.59:0/3423761289 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed8c102500 msgr2=0x7fed8c111460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:24.002 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.425+0000 7fed9379d700 1 --2- 172.21.15.59:0/3423761289 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed8c102500 0x7fed8c111460 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fed80000c00 tx=0x7fed80009250 comp rx=0 tx=0).stop 2024-01-23T08:43:24.002 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.425+0000 7fed9379d700 1 -- 172.21.15.59:0/3423761289 shutdown_connections 2024-01-23T08:43:24.002 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.425+0000 7fed9379d700 1 --2- 172.21.15.59:0/3423761289 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7fed780365a0 0x7fed78038a50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:24.002 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.425+0000 7fed9379d700 1 --2- 172.21.15.59:0/3423761289 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed8c102500 0x7fed8c111460 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:24.002 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.425+0000 7fed9379d700 1 -- 172.21.15.59:0/3423761289 >> 172.21.15.59:0/3423761289 conn(0x7fed8c0fdc80 msgr2=0x7fed8c0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:24.002 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.426+0000 7fed9379d700 1 -- 172.21.15.59:0/3423761289 shutdown_connections 2024-01-23T08:43:24.002 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:23.426+0000 7fed9379d700 1 -- 172.21.15.59:0/3423761289 wait complete. 2024-01-23T08:43:24.002 INFO:teuthology.orchestra.run.smithi059.stdout:Enabling cephadm module... 2024-01-23T08:43:25.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:24 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/3321083943' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-01-23T08:43:25.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.430+0000 7fcf71b23700 1 Processor -- start 2024-01-23T08:43:25.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.430+0000 7fcf71b23700 1 -- start start 2024-01-23T08:43:25.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.430+0000 7fcf71b23700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6c102450 0x7fcf6c102860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:25.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.430+0000 7fcf71b23700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c102da0 con 0x7fcf6c102450 2024-01-23T08:43:25.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.430+0000 7fcf6b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6c102450 0x7fcf6c102860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:25.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.430+0000 7fcf6b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6c102450 0x7fcf6c102860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56290/0 (socket says 172.21.15.59:56290) 2024-01-23T08:43:25.544 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.430+0000 7fcf6b7fe700 1 -- 172.21.15.59:0/3270149855 learned_addr learned my addr 172.21.15.59:0/3270149855 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:25.544 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.430+0000 7fcf6b7fe700 1 -- 172.21.15.59:0/3270149855 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf6c102ee0 con 0x7fcf6c102450 2024-01-23T08:43:25.544 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.430+0000 7fcf6b7fe700 1 --2- 172.21.15.59:0/3270149855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6c102450 0x7fcf6c102860 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c0098e0 tx=0x7fcf5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9bb4d11c898706 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:25.544 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.430+0000 7fcf6a7fc700 1 -- 172.21.15.59:0/3270149855 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf5c00b860 con 0x7fcf6c102450 2024-01-23T08:43:25.544 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.430+0000 7fcf6a7fc700 1 -- 172.21.15.59:0/3270149855 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fcf5c00be30 con 0x7fcf6c102450 2024-01-23T08:43:25.544 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.430+0000 7fcf6a7fc700 1 -- 172.21.15.59:0/3270149855 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf5c010380 con 0x7fcf6c102450 2024-01-23T08:43:25.544 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.431+0000 7fcf71b23700 1 -- 172.21.15.59:0/3270149855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6c102450 msgr2=0x7fcf6c102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:25.544 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.431+0000 7fcf71b23700 1 --2- 172.21.15.59:0/3270149855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6c102450 0x7fcf6c102860 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c0098e0 tx=0x7fcf5c009250 comp rx=0 tx=0).stop 2024-01-23T08:43:25.544 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.431+0000 7fcf71b23700 1 -- 172.21.15.59:0/3270149855 shutdown_connections 2024-01-23T08:43:25.544 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.431+0000 7fcf71b23700 1 --2- 172.21.15.59:0/3270149855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6c102450 0x7fcf6c102860 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:25.544 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.431+0000 7fcf71b23700 1 -- 172.21.15.59:0/3270149855 >> 172.21.15.59:0/3270149855 conn(0x7fcf6c0fdbd0 msgr2=0x7fcf6c0fffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:25.544 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.431+0000 7fcf71b23700 1 -- 172.21.15.59:0/3270149855 shutdown_connections 2024-01-23T08:43:25.544 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.431+0000 7fcf71b23700 1 -- 172.21.15.59:0/3270149855 wait complete. 2024-01-23T08:43:25.545 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.432+0000 7fcf71b23700 1 Processor -- start 2024-01-23T08:43:25.545 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.432+0000 7fcf71b23700 1 -- start start 2024-01-23T08:43:25.545 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.432+0000 7fcf71b23700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6c102450 0x7fcf6c078b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:25.545 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.432+0000 7fcf71b23700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c079090 con 0x7fcf6c102450 2024-01-23T08:43:25.545 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.432+0000 7fcf6b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6c102450 0x7fcf6c078b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:25.545 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.432+0000 7fcf6b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6c102450 0x7fcf6c078b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56306/0 (socket says 172.21.15.59:56306) 2024-01-23T08:43:25.545 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.432+0000 7fcf6b7fe700 1 -- 172.21.15.59:0/3321083943 learned_addr learned my addr 172.21.15.59:0/3321083943 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:25.545 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.432+0000 7fcf6b7fe700 1 -- 172.21.15.59:0/3321083943 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf5c0094f0 con 0x7fcf6c102450 2024-01-23T08:43:25.545 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.433+0000 7fcf6b7fe700 1 --2- 172.21.15.59:0/3321083943 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6c102450 0x7fcf6c078b50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c000c00 tx=0x7fcf5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:25.545 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.433+0000 7fcf63fff700 1 -- 172.21.15.59:0/3321083943 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf5c010a80 con 0x7fcf6c102450 2024-01-23T08:43:25.545 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.433+0000 7fcf71b23700 1 -- 172.21.15.59:0/3321083943 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf6c0771b0 con 0x7fcf6c102450 2024-01-23T08:43:25.545 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.433+0000 7fcf63fff700 1 -- 172.21.15.59:0/3321083943 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fcf5c01a430 con 0x7fcf6c102450 2024-01-23T08:43:25.545 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.433+0000 7fcf63fff700 1 -- 172.21.15.59:0/3321083943 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf5c0245b0 con 0x7fcf6c102450 2024-01-23T08:43:25.545 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.433+0000 7fcf71b23700 1 -- 172.21.15.59:0/3321083943 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf6c077620 con 0x7fcf6c102450 2024-01-23T08:43:25.545 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.434+0000 7fcf63fff700 1 -- 172.21.15.59:0/3321083943 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 4) v1 ==== 43270+0+0 (secure 0 0 0) 0x7fcf5c024870 con 0x7fcf6c102450 2024-01-23T08:43:25.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.434+0000 7fcf63fff700 1 --2- 172.21.15.59:0/3321083943 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7fcf580365a0 0x7fcf58038a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:25.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.434+0000 7fcf63fff700 1 -- 172.21.15.59:0/3321083943 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fcf5c04a670 con 0x7fcf6c102450 2024-01-23T08:43:25.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.434+0000 7fcf6affd700 1 --2- 172.21.15.59:0/3321083943 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7fcf580365a0 0x7fcf58038a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:25.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.434+0000 7fcf71b23700 1 -- 172.21.15.59:0/3321083943 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf6c1042d0 con 0x7fcf6c102450 2024-01-23T08:43:25.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.434+0000 7fcf6affd700 1 --2- 172.21.15.59:0/3321083943 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7fcf580365a0 0x7fcf58038a50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcf54009990 tx=0x7fcf54006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:25.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.439+0000 7fcf63fff700 1 -- 172.21.15.59:0/3321083943 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fcf5c02c340 con 0x7fcf6c102450 2024-01-23T08:43:25.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.685+0000 7fcf71b23700 1 -- 172.21.15.59:0/3321083943 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fcf6c077bc0 con 0x7fcf6c102450 2024-01-23T08:43:25.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.974+0000 7fcf63fff700 1 -- 172.21.15.59:0/3321083943 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 5) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fcf5c02e960 con 0x7fcf6c102450 2024-01-23T08:43:25.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.975+0000 7fcf63fff700 1 -- 172.21.15.59:0/3321083943 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fcf5c00eb00 con 0x7fcf6c102450 2024-01-23T08:43:25.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.977+0000 7fcf71b23700 1 -- 172.21.15.59:0/3321083943 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7fcf580365a0 msgr2=0x7fcf58038a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:25.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.977+0000 7fcf71b23700 1 --2- 172.21.15.59:0/3321083943 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7fcf580365a0 0x7fcf58038a50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcf54009990 tx=0x7fcf54006e40 comp rx=0 tx=0).stop 2024-01-23T08:43:25.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.977+0000 7fcf71b23700 1 -- 172.21.15.59:0/3321083943 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6c102450 msgr2=0x7fcf6c078b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:25.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.977+0000 7fcf71b23700 1 --2- 172.21.15.59:0/3321083943 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6c102450 0x7fcf6c078b50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c000c00 tx=0x7fcf5c009250 comp rx=0 tx=0).stop 2024-01-23T08:43:25.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.977+0000 7fcf71b23700 1 -- 172.21.15.59:0/3321083943 shutdown_connections 2024-01-23T08:43:25.546 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.977+0000 7fcf71b23700 1 --2- 172.21.15.59:0/3321083943 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7fcf580365a0 0x7fcf58038a50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:25.547 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.977+0000 7fcf71b23700 1 --2- 172.21.15.59:0/3321083943 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6c102450 0x7fcf6c078b50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:25.547 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.977+0000 7fcf71b23700 1 -- 172.21.15.59:0/3321083943 >> 172.21.15.59:0/3321083943 conn(0x7fcf6c0fdbd0 msgr2=0x7fcf6c0ff2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:25.547 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.977+0000 7fcf71b23700 1 -- 172.21.15.59:0/3321083943 shutdown_connections 2024-01-23T08:43:25.547 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:24.977+0000 7fcf71b23700 1 -- 172.21.15.59:0/3321083943 wait complete. 2024-01-23T08:43:26.164 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:25 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/3321083943' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-01-23T08:43:26.164 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:25 smithi059 ceph-mon[28424]: mgrmap e5: smithi059.bzqirf(active, since 6s) 2024-01-23T08:43:26.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout { 2024-01-23T08:43:26.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-01-23T08:43:26.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "available": true, 2024-01-23T08:43:26.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "active_name": "smithi059.bzqirf", 2024-01-23T08:43:26.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-01-23T08:43:26.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout } 2024-01-23T08:43:26.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.961+0000 7fcb555a7700 1 Processor -- start 2024-01-23T08:43:26.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.961+0000 7fcb555a7700 1 -- start start 2024-01-23T08:43:26.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.961+0000 7fcb555a7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb50102ab0 0x7fcb50102ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:26.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.961+0000 7fcb555a7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb50103400 con 0x7fcb50102ab0 2024-01-23T08:43:26.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.961+0000 7fcb4effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb50102ab0 0x7fcb50102ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:26.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.961+0000 7fcb4effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb50102ab0 0x7fcb50102ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56334/0 (socket says 172.21.15.59:56334) 2024-01-23T08:43:26.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.961+0000 7fcb4effd700 1 -- 172.21.15.59:0/110211479 learned_addr learned my addr 172.21.15.59:0/110211479 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:26.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.961+0000 7fcb4effd700 1 -- 172.21.15.59:0/110211479 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb50103540 con 0x7fcb50102ab0 2024-01-23T08:43:26.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.962+0000 7fcb4effd700 1 --2- 172.21.15.59:0/110211479 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb50102ab0 0x7fcb50102ec0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcb400098e0 tx=0x7fcb40009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f34d2efcd37941a server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:26.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.962+0000 7fcb4dffb700 1 -- 172.21.15.59:0/110211479 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb4000b860 con 0x7fcb50102ab0 2024-01-23T08:43:26.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.962+0000 7fcb4dffb700 1 -- 172.21.15.59:0/110211479 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fcb4000be30 con 0x7fcb50102ab0 2024-01-23T08:43:26.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.962+0000 7fcb4dffb700 1 -- 172.21.15.59:0/110211479 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb40010380 con 0x7fcb50102ab0 2024-01-23T08:43:26.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.962+0000 7fcb555a7700 1 -- 172.21.15.59:0/110211479 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb50102ab0 msgr2=0x7fcb50102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:26.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.962+0000 7fcb555a7700 1 --2- 172.21.15.59:0/110211479 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb50102ab0 0x7fcb50102ec0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcb400098e0 tx=0x7fcb40009250 comp rx=0 tx=0).stop 2024-01-23T08:43:26.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.963+0000 7fcb555a7700 1 -- 172.21.15.59:0/110211479 shutdown_connections 2024-01-23T08:43:26.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.963+0000 7fcb555a7700 1 --2- 172.21.15.59:0/110211479 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb50102ab0 0x7fcb50102ec0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:26.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.963+0000 7fcb555a7700 1 -- 172.21.15.59:0/110211479 >> 172.21.15.59:0/110211479 conn(0x7fcb500fc260 msgr2=0x7fcb500fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:26.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.963+0000 7fcb555a7700 1 -- 172.21.15.59:0/110211479 shutdown_connections 2024-01-23T08:43:26.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.963+0000 7fcb555a7700 1 -- 172.21.15.59:0/110211479 wait complete. 2024-01-23T08:43:26.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.963+0000 7fcb555a7700 1 Processor -- start 2024-01-23T08:43:26.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.963+0000 7fcb555a7700 1 -- start start 2024-01-23T08:43:26.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.964+0000 7fcb555a7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb50102ab0 0x7fcb50113400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:26.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.964+0000 7fcb555a7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb50113940 con 0x7fcb50102ab0 2024-01-23T08:43:26.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.964+0000 7fcb4effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb50102ab0 0x7fcb50113400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:26.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.964+0000 7fcb4effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb50102ab0 0x7fcb50113400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56350/0 (socket says 172.21.15.59:56350) 2024-01-23T08:43:26.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.964+0000 7fcb4effd700 1 -- 172.21.15.59:0/738263687 learned_addr learned my addr 172.21.15.59:0/738263687 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:26.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.964+0000 7fcb4effd700 1 -- 172.21.15.59:0/738263687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb400094f0 con 0x7fcb50102ab0 2024-01-23T08:43:26.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.964+0000 7fcb4effd700 1 --2- 172.21.15.59:0/738263687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb50102ab0 0x7fcb50113400 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcb40000c00 tx=0x7fcb40009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:26.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.964+0000 7fcb377fe700 1 -- 172.21.15.59:0/738263687 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb40010a80 con 0x7fcb50102ab0 2024-01-23T08:43:26.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.964+0000 7fcb377fe700 1 -- 172.21.15.59:0/738263687 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fcb4001a430 con 0x7fcb50102ab0 2024-01-23T08:43:26.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.964+0000 7fcb555a7700 1 -- 172.21.15.59:0/738263687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb50113b40 con 0x7fcb50102ab0 2024-01-23T08:43:26.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.964+0000 7fcb377fe700 1 -- 172.21.15.59:0/738263687 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb400245b0 con 0x7fcb50102ab0 2024-01-23T08:43:26.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.965+0000 7fcb555a7700 1 -- 172.21.15.59:0/738263687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb50113f50 con 0x7fcb50102ab0 2024-01-23T08:43:26.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.965+0000 7fcb377fe700 1 -- 172.21.15.59:0/738263687 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fcb400247a0 con 0x7fcb50102ab0 2024-01-23T08:43:26.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.966+0000 7fcb377fe700 1 --2- 172.21.15.59:0/738263687 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7fcb38036640 0x7fcb38038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:26.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.966+0000 7fcb555a7700 1 -- 172.21.15.59:0/738263687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb501087e0 con 0x7fcb50102ab0 2024-01-23T08:43:26.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.966+0000 7fcb377fe700 1 -- 172.21.15.59:0/738263687 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fcb4004ba90 con 0x7fcb50102ab0 2024-01-23T08:43:26.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.966+0000 7fcb4e7fc700 1 -- 172.21.15.59:0/738263687 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7fcb38036640 msgr2=0x7fcb38038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/504555577 2024-01-23T08:43:26.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.966+0000 7fcb4e7fc700 1 --2- 172.21.15.59:0/738263687 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7fcb38036640 0x7fcb38038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-23T08:43:26.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:25.970+0000 7fcb377fe700 1 -- 172.21.15.59:0/738263687 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fcb40018350 con 0x7fcb50102ab0 2024-01-23T08:43:26.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:26.167+0000 7fcb4e7fc700 1 -- 172.21.15.59:0/738263687 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7fcb38036640 msgr2=0x7fcb38038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/504555577 2024-01-23T08:43:26.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:26.167+0000 7fcb4e7fc700 1 --2- 172.21.15.59:0/738263687 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7fcb38036640 0x7fcb38038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-23T08:43:26.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:26.222+0000 7fcb555a7700 1 -- 172.21.15.59:0/738263687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fcb50116d90 con 0x7fcb50102ab0 2024-01-23T08:43:26.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:26.222+0000 7fcb377fe700 1 -- 172.21.15.59:0/738263687 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fcb40023b80 con 0x7fcb50102ab0 2024-01-23T08:43:26.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:26.224+0000 7fcb555a7700 1 -- 172.21.15.59:0/738263687 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7fcb38036640 msgr2=0x7fcb38038af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-23T08:43:26.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:26.224+0000 7fcb555a7700 1 --2- 172.21.15.59:0/738263687 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7fcb38036640 0x7fcb38038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:26.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:26.224+0000 7fcb555a7700 1 -- 172.21.15.59:0/738263687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb50102ab0 msgr2=0x7fcb50113400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:26.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:26.224+0000 7fcb555a7700 1 --2- 172.21.15.59:0/738263687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb50102ab0 0x7fcb50113400 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcb40000c00 tx=0x7fcb40009250 comp rx=0 tx=0).stop 2024-01-23T08:43:26.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:26.224+0000 7fcb555a7700 1 -- 172.21.15.59:0/738263687 shutdown_connections 2024-01-23T08:43:26.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:26.224+0000 7fcb555a7700 1 --2- 172.21.15.59:0/738263687 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7fcb38036640 0x7fcb38038af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:26.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:26.224+0000 7fcb555a7700 1 --2- 172.21.15.59:0/738263687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb50102ab0 0x7fcb50113400 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:26.837 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:26.224+0000 7fcb555a7700 1 -- 172.21.15.59:0/738263687 >> 172.21.15.59:0/738263687 conn(0x7fcb500fc260 msgr2=0x7fcb500fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:26.838 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:26.224+0000 7fcb555a7700 1 -- 172.21.15.59:0/738263687 shutdown_connections 2024-01-23T08:43:26.838 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:26.225+0000 7fcb555a7700 1 -- 172.21.15.59:0/738263687 wait complete. 2024-01-23T08:43:26.838 INFO:teuthology.orchestra.run.smithi059.stdout:Waiting for the mgr to restart... 2024-01-23T08:43:26.838 INFO:teuthology.orchestra.run.smithi059.stdout:Waiting for mgr epoch 5... 2024-01-23T08:43:27.079 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:26 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/738263687' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-23T08:43:31.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:31 smithi059 ceph-mon[28424]: Active manager daemon smithi059.bzqirf restarted 2024-01-23T08:43:31.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:31 smithi059 ceph-mon[28424]: Activating manager daemon smithi059.bzqirf 2024-01-23T08:43:31.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:31 smithi059 ceph-mon[28424]: osdmap e2: 0 total, 0 up, 0 in 2024-01-23T08:43:31.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:31 smithi059 ceph-mon[28424]: mgrmap e6: smithi059.bzqirf(active, starting, since 0.0033946s) 2024-01-23T08:43:31.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:31 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-01-23T08:43:31.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:31 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mgr metadata", "who": "smithi059.bzqirf", "id": "smithi059.bzqirf"}]: dispatch 2024-01-23T08:43:31.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:31 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-23T08:43:31.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:31 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-23T08:43:31.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:31 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-23T08:43:31.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:31 smithi059 ceph-mon[28424]: Manager daemon smithi059.bzqirf is now available 2024-01-23T08:43:31.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:31 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' 2024-01-23T08:43:31.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:31 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' 2024-01-23T08:43:31.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:31 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:43:31.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:31 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:43:31.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:31 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:43:32.964 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:32 smithi059 ceph-mon[28424]: Found migration_current of "None". Setting to last migration. 2024-01-23T08:43:32.964 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:32 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.bzqirf/mirror_snapshot_schedule"}]: dispatch 2024-01-23T08:43:32.965 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:32 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.bzqirf/trash_purge_schedule"}]: dispatch 2024-01-23T08:43:32.965 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:32 smithi059 ceph-mon[28424]: mgrmap e7: smithi059.bzqirf(active, since 1.00524s) 2024-01-23T08:43:33.232 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout { 2024-01-23T08:43:33.232 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-01-23T08:43:33.232 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "initialized": true 2024-01-23T08:43:33.233 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout } 2024-01-23T08:43:33.233 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.277+0000 7f32adcde700 1 Processor -- start 2024-01-23T08:43:33.233 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.277+0000 7f32adcde700 1 -- start start 2024-01-23T08:43:33.233 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.277+0000 7f32adcde700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32a8101400 0x7f32a8101810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:33.233 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.277+0000 7f32adcde700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a8101de0 con 0x7f32a8101400 2024-01-23T08:43:33.233 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.278+0000 7f32a77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32a8101400 0x7f32a8101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:33.233 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.278+0000 7f32a77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32a8101400 0x7f32a8101810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56360/0 (socket says 172.21.15.59:56360) 2024-01-23T08:43:33.233 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.278+0000 7f32a77fe700 1 -- 172.21.15.59:0/747449645 learned_addr learned my addr 172.21.15.59:0/747449645 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:33.233 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.278+0000 7f32a77fe700 1 -- 172.21.15.59:0/747449645 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a8102610 con 0x7f32a8101400 2024-01-23T08:43:33.233 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.278+0000 7f32a77fe700 1 --2- 172.21.15.59:0/747449645 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32a8101400 0x7f32a8101810 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f32980098e0 tx=0x7f3298009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a97f2c691286fde server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:33.235 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.278+0000 7f32a67fc700 1 -- 172.21.15.59:0/747449645 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f329800b860 con 0x7f32a8101400 2024-01-23T08:43:33.235 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.278+0000 7f32a67fc700 1 -- 172.21.15.59:0/747449645 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f329800be30 con 0x7f32a8101400 2024-01-23T08:43:33.235 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.278+0000 7f32a67fc700 1 -- 172.21.15.59:0/747449645 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3298010380 con 0x7f32a8101400 2024-01-23T08:43:33.235 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.279+0000 7f32adcde700 1 -- 172.21.15.59:0/747449645 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32a8101400 msgr2=0x7f32a8101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:33.235 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.279+0000 7f32adcde700 1 --2- 172.21.15.59:0/747449645 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32a8101400 0x7f32a8101810 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f32980098e0 tx=0x7f3298009250 comp rx=0 tx=0).stop 2024-01-23T08:43:33.235 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.279+0000 7f32adcde700 1 -- 172.21.15.59:0/747449645 shutdown_connections 2024-01-23T08:43:33.235 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.279+0000 7f32adcde700 1 --2- 172.21.15.59:0/747449645 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32a8101400 0x7f32a8101810 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:33.235 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.279+0000 7f32adcde700 1 -- 172.21.15.59:0/747449645 >> 172.21.15.59:0/747449645 conn(0x7f32a80fcae0 msgr2=0x7f32a80feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:33.235 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.279+0000 7f32adcde700 1 -- 172.21.15.59:0/747449645 shutdown_connections 2024-01-23T08:43:33.235 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.279+0000 7f32adcde700 1 -- 172.21.15.59:0/747449645 wait complete. 2024-01-23T08:43:33.235 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.280+0000 7f32adcde700 1 Processor -- start 2024-01-23T08:43:33.235 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.280+0000 7f32adcde700 1 -- start start 2024-01-23T08:43:33.235 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.280+0000 7f32adcde700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32a8101400 0x7f32a81100f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:33.235 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.280+0000 7f32adcde700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a8110630 con 0x7f32a8101400 2024-01-23T08:43:33.236 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.280+0000 7f32a77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32a8101400 0x7f32a81100f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:33.236 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.280+0000 7f32a77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32a8101400 0x7f32a81100f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56364/0 (socket says 172.21.15.59:56364) 2024-01-23T08:43:33.236 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.280+0000 7f32a77fe700 1 -- 172.21.15.59:0/1593402681 learned_addr learned my addr 172.21.15.59:0/1593402681 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:33.236 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.280+0000 7f32a77fe700 1 -- 172.21.15.59:0/1593402681 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32980094f0 con 0x7f32a8101400 2024-01-23T08:43:33.236 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.280+0000 7f32a77fe700 1 --2- 172.21.15.59:0/1593402681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32a8101400 0x7f32a81100f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f32980090b0 tx=0x7f3298009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:33.236 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.281+0000 7f329ffff700 1 -- 172.21.15.59:0/1593402681 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3298010a80 con 0x7f32a8101400 2024-01-23T08:43:33.236 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.281+0000 7f329ffff700 1 -- 172.21.15.59:0/1593402681 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f329801a430 con 0x7f32a8101400 2024-01-23T08:43:33.236 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.281+0000 7f32adcde700 1 -- 172.21.15.59:0/1593402681 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a8110830 con 0x7f32a8101400 2024-01-23T08:43:33.236 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.281+0000 7f329ffff700 1 -- 172.21.15.59:0/1593402681 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32980245b0 con 0x7f32a8101400 2024-01-23T08:43:33.236 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.281+0000 7f32adcde700 1 -- 172.21.15.59:0/1593402681 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a8110c40 con 0x7f32a8101400 2024-01-23T08:43:33.236 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.282+0000 7f329ffff700 1 -- 172.21.15.59:0/1593402681 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f329802d3f0 con 0x7f32a8101400 2024-01-23T08:43:33.236 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.282+0000 7f329ffff700 1 --2- 172.21.15.59:0/1593402681 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7f32940365f0 0x7f3294038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:33.236 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.282+0000 7f329ffff700 1 -- 172.21.15.59:0/1593402681 --> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f32940391b0 con 0x7f32940365f0 2024-01-23T08:43:33.236 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.282+0000 7f329ffff700 1 -- 172.21.15.59:0/1593402681 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f329804a190 con 0x7f32a8101400 2024-01-23T08:43:33.237 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.282+0000 7f32a6ffd700 1 -- 172.21.15.59:0/1593402681 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7f32940365f0 msgr2=0x7f3294038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/504555577 2024-01-23T08:43:33.237 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.282+0000 7f32a6ffd700 1 --2- 172.21.15.59:0/1593402681 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7f32940365f0 0x7f3294038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-23T08:43:33.237 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.483+0000 7f32a6ffd700 1 -- 172.21.15.59:0/1593402681 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7f32940365f0 msgr2=0x7f3294038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/504555577 2024-01-23T08:43:33.237 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.483+0000 7f32a6ffd700 1 --2- 172.21.15.59:0/1593402681 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7f32940365f0 0x7f3294038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-23T08:43:33.237 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.884+0000 7f32a6ffd700 1 -- 172.21.15.59:0/1593402681 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7f32940365f0 msgr2=0x7f3294038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/504555577 2024-01-23T08:43:33.237 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:27.884+0000 7f32a6ffd700 1 --2- 172.21.15.59:0/1593402681 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7f32940365f0 0x7f3294038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-23T08:43:33.237 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:28.685+0000 7f32a6ffd700 1 -- 172.21.15.59:0/1593402681 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7f32940365f0 msgr2=0x7f3294038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/504555577 2024-01-23T08:43:33.237 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:28.685+0000 7f32a6ffd700 1 --2- 172.21.15.59:0/1593402681 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7f32940365f0 0x7f3294038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-23T08:43:33.237 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:30.286+0000 7f32a6ffd700 1 -- 172.21.15.59:0/1593402681 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7f32940365f0 msgr2=0x7f3294038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/504555577 2024-01-23T08:43:33.237 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:30.286+0000 7f32a6ffd700 1 --2- 172.21.15.59:0/1593402681 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7f32940365f0 0x7f3294038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-23T08:43:33.237 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:31.662+0000 7f329ffff700 1 -- 172.21.15.59:0/1593402681 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mgrmap(e 6) v1 ==== 43048+0+0 (secure 0 0 0) 0x7f329802d6f0 con 0x7f32a8101400 2024-01-23T08:43:33.237 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:31.662+0000 7f329ffff700 1 -- 172.21.15.59:0/1593402681 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7f32940365f0 msgr2=0x7f3294038aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-23T08:43:33.237 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:31.662+0000 7f329ffff700 1 --2- 172.21.15.59:0/1593402681 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7f32940365f0 0x7f3294038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:33.237 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:32.663+0000 7f329ffff700 1 -- 172.21.15.59:0/1593402681 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 7) v1 ==== 43175+0+0 (secure 0 0 0) 0x7f329802da70 con 0x7f32a8101400 2024-01-23T08:43:33.237 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:32.663+0000 7f329ffff700 1 --2- 172.21.15.59:0/1593402681 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f3294039f60 0x7f329403c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:33.237 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:32.663+0000 7f329ffff700 1 -- 172.21.15.59:0/1593402681 --> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f32940391b0 con 0x7f3294039f60 2024-01-23T08:43:33.238 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:32.664+0000 7f32a6ffd700 1 --2- 172.21.15.59:0/1593402681 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f3294039f60 0x7f329403c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:33.238 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:32.664+0000 7f32a6ffd700 1 --2- 172.21.15.59:0/1593402681 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f3294039f60 0x7f329403c340 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f32900039d0 tx=0x7f3290009350 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:33.238 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:32.665+0000 7f329ffff700 1 -- 172.21.15.59:0/1593402681 <== mgr.14120 v2:172.21.15.59:6800/1337467262 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f32940391b0 con 0x7f3294039f60 2024-01-23T08:43:33.238 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:32.671+0000 7f32adcde700 1 -- 172.21.15.59:0/1593402681 --> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f32a805f450 con 0x7f3294039f60 2024-01-23T08:43:33.238 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:32.671+0000 7f329ffff700 1 -- 172.21.15.59:0/1593402681 <== mgr.14120 v2:172.21.15.59:6800/1337467262 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f32a805f450 con 0x7f3294039f60 2024-01-23T08:43:33.238 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:32.671+0000 7f32adcde700 1 -- 172.21.15.59:0/1593402681 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f3294039f60 msgr2=0x7f329403c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:33.238 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:32.671+0000 7f32adcde700 1 --2- 172.21.15.59:0/1593402681 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f3294039f60 0x7f329403c340 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f32900039d0 tx=0x7f3290009350 comp rx=0 tx=0).stop 2024-01-23T08:43:33.238 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:32.671+0000 7f32adcde700 1 -- 172.21.15.59:0/1593402681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32a8101400 msgr2=0x7f32a81100f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:33.238 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:32.671+0000 7f32adcde700 1 --2- 172.21.15.59:0/1593402681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32a8101400 0x7f32a81100f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f32980090b0 tx=0x7f3298009250 comp rx=0 tx=0).stop 2024-01-23T08:43:33.238 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:32.671+0000 7f32adcde700 1 -- 172.21.15.59:0/1593402681 shutdown_connections 2024-01-23T08:43:33.238 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:32.671+0000 7f32adcde700 1 --2- 172.21.15.59:0/1593402681 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f3294039f60 0x7f329403c340 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:33.238 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:32.671+0000 7f32adcde700 1 --2- 172.21.15.59:0/1593402681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32a8101400 0x7f32a81100f0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:33.238 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:32.671+0000 7f32adcde700 1 --2- 172.21.15.59:0/1593402681 >> [v2:172.21.15.59:6800/504555577,v1:172.21.15.59:6801/504555577] conn(0x7f32940365f0 0x7f3294038aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:33.238 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:32.671+0000 7f32adcde700 1 -- 172.21.15.59:0/1593402681 >> 172.21.15.59:0/1593402681 conn(0x7f32a80fcae0 msgr2=0x7f32a80fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:33.238 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:32.672+0000 7f32adcde700 1 -- 172.21.15.59:0/1593402681 shutdown_connections 2024-01-23T08:43:33.239 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:32.672+0000 7f32adcde700 1 -- 172.21.15.59:0/1593402681 wait complete. 2024-01-23T08:43:33.239 INFO:teuthology.orchestra.run.smithi059.stdout:mgr epoch 5 is available 2024-01-23T08:43:33.239 INFO:teuthology.orchestra.run.smithi059.stdout:Setting orchestrator backend to cephadm... 2024-01-23T08:43:33.836 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:33 smithi059 ceph-mon[28424]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-23T08:43:33.837 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:33 smithi059 ceph-mon[28424]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-23T08:43:34.503 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.607+0000 7f7b6caac700 1 Processor -- start 2024-01-23T08:43:34.503 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.608+0000 7f7b6caac700 1 -- start start 2024-01-23T08:43:34.503 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.608+0000 7f7b6caac700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b68101400 0x7f7b68101810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:34.503 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.608+0000 7f7b6caac700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b68101de0 con 0x7f7b68101400 2024-01-23T08:43:34.503 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.608+0000 7f7b6659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b68101400 0x7f7b68101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:34.503 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.608+0000 7f7b6659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b68101400 0x7f7b68101810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49384/0 (socket says 172.21.15.59:49384) 2024-01-23T08:43:34.503 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.608+0000 7f7b6659c700 1 -- 172.21.15.59:0/3365161563 learned_addr learned my addr 172.21.15.59:0/3365161563 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:34.504 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.608+0000 7f7b6659c700 1 -- 172.21.15.59:0/3365161563 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b68102610 con 0x7f7b68101400 2024-01-23T08:43:34.504 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.609+0000 7f7b6659c700 1 --2- 172.21.15.59:0/3365161563 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b68101400 0x7f7b68101810 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7b580098e0 tx=0x7f7b58009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c30fb9206e4a3bd2 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:34.504 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.609+0000 7f7b6559a700 1 -- 172.21.15.59:0/3365161563 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b5800b860 con 0x7f7b68101400 2024-01-23T08:43:34.504 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.609+0000 7f7b6559a700 1 -- 172.21.15.59:0/3365161563 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f7b5800be30 con 0x7f7b68101400 2024-01-23T08:43:34.504 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.609+0000 7f7b6559a700 1 -- 172.21.15.59:0/3365161563 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b58010380 con 0x7f7b68101400 2024-01-23T08:43:34.504 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.609+0000 7f7b6caac700 1 -- 172.21.15.59:0/3365161563 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b68101400 msgr2=0x7f7b68101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:34.504 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.609+0000 7f7b6caac700 1 --2- 172.21.15.59:0/3365161563 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b68101400 0x7f7b68101810 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7b580098e0 tx=0x7f7b58009250 comp rx=0 tx=0).stop 2024-01-23T08:43:34.504 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.610+0000 7f7b6caac700 1 -- 172.21.15.59:0/3365161563 shutdown_connections 2024-01-23T08:43:34.504 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.610+0000 7f7b6caac700 1 --2- 172.21.15.59:0/3365161563 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b68101400 0x7f7b68101810 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:34.504 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.610+0000 7f7b6caac700 1 -- 172.21.15.59:0/3365161563 >> 172.21.15.59:0/3365161563 conn(0x7f7b680fcae0 msgr2=0x7f7b680feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:34.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.610+0000 7f7b6caac700 1 -- 172.21.15.59:0/3365161563 shutdown_connections 2024-01-23T08:43:34.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.610+0000 7f7b6caac700 1 -- 172.21.15.59:0/3365161563 wait complete. 2024-01-23T08:43:34.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.610+0000 7f7b6caac700 1 Processor -- start 2024-01-23T08:43:34.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.610+0000 7f7b6caac700 1 -- start start 2024-01-23T08:43:34.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.610+0000 7f7b6caac700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b68101400 0x7f7b68076a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:34.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.610+0000 7f7b6caac700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b68113fe0 con 0x7f7b68101400 2024-01-23T08:43:34.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.610+0000 7f7b6659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b68101400 0x7f7b68076a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:34.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.610+0000 7f7b6659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b68101400 0x7f7b68076a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49388/0 (socket says 172.21.15.59:49388) 2024-01-23T08:43:34.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.610+0000 7f7b6659c700 1 -- 172.21.15.59:0/2204467813 learned_addr learned my addr 172.21.15.59:0/2204467813 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:34.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.611+0000 7f7b6659c700 1 -- 172.21.15.59:0/2204467813 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b580094f0 con 0x7f7b68101400 2024-01-23T08:43:34.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.611+0000 7f7b6659c700 1 --2- 172.21.15.59:0/2204467813 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b68101400 0x7f7b68076a30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7b580090b0 tx=0x7f7b58009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:34.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.611+0000 7f7b56ffd700 1 -- 172.21.15.59:0/2204467813 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b580109b0 con 0x7f7b68101400 2024-01-23T08:43:34.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.611+0000 7f7b56ffd700 1 -- 172.21.15.59:0/2204467813 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f7b5801a430 con 0x7f7b68101400 2024-01-23T08:43:34.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.611+0000 7f7b6caac700 1 -- 172.21.15.59:0/2204467813 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b68114120 con 0x7f7b68101400 2024-01-23T08:43:34.505 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.611+0000 7f7b56ffd700 1 -- 172.21.15.59:0/2204467813 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b580245b0 con 0x7f7b68101400 2024-01-23T08:43:34.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.611+0000 7f7b6caac700 1 -- 172.21.15.59:0/2204467813 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b68114470 con 0x7f7b68101400 2024-01-23T08:43:34.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.612+0000 7f7b56ffd700 1 -- 172.21.15.59:0/2204467813 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 7) v1 ==== 43175+0+0 (secure 0 0 0) 0x7f7b58010b10 con 0x7f7b68101400 2024-01-23T08:43:34.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.612+0000 7f7b56ffd700 1 --2- 172.21.15.59:0/2204467813 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f7b500364c0 0x7f7b50038970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:34.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.612+0000 7f7b56ffd700 1 -- 172.21.15.59:0/2204467813 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f7b5804a490 con 0x7f7b68101400 2024-01-23T08:43:34.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.612+0000 7f7b65d9b700 1 --2- 172.21.15.59:0/2204467813 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f7b500364c0 0x7f7b50038970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:34.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.612+0000 7f7b6caac700 1 -- 172.21.15.59:0/2204467813 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b6805f450 con 0x7f7b68101400 2024-01-23T08:43:34.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.613+0000 7f7b65d9b700 1 --2- 172.21.15.59:0/2204467813 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f7b500364c0 0x7f7b50038970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c009990 tx=0x7f7b5c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:34.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.617+0000 7f7b56ffd700 1 -- 172.21.15.59:0/2204467813 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7b5802e440 con 0x7f7b68101400 2024-01-23T08:43:34.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.714+0000 7f7b56ffd700 1 -- 172.21.15.59:0/2204467813 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f7b58024d40 con 0x7f7b68101400 2024-01-23T08:43:34.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.839+0000 7f7b6caac700 1 -- 172.21.15.59:0/2204467813 --> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b680fee20 con 0x7f7b500364c0 2024-01-23T08:43:34.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.846+0000 7f7b56ffd700 1 -- 172.21.15.59:0/2204467813 <== mgr.14120 v2:172.21.15.59:6800/1337467262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7b680fee20 con 0x7f7b500364c0 2024-01-23T08:43:34.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.848+0000 7f7b6caac700 1 -- 172.21.15.59:0/2204467813 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f7b500364c0 msgr2=0x7f7b50038970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:34.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.848+0000 7f7b6caac700 1 --2- 172.21.15.59:0/2204467813 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f7b500364c0 0x7f7b50038970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c009990 tx=0x7f7b5c006e40 comp rx=0 tx=0).stop 2024-01-23T08:43:34.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.849+0000 7f7b6caac700 1 -- 172.21.15.59:0/2204467813 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b68101400 msgr2=0x7f7b68076a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:34.506 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.849+0000 7f7b6caac700 1 --2- 172.21.15.59:0/2204467813 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b68101400 0x7f7b68076a30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7b580090b0 tx=0x7f7b58009250 comp rx=0 tx=0).stop 2024-01-23T08:43:34.507 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.849+0000 7f7b6caac700 1 -- 172.21.15.59:0/2204467813 shutdown_connections 2024-01-23T08:43:34.507 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.849+0000 7f7b6caac700 1 --2- 172.21.15.59:0/2204467813 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f7b500364c0 0x7f7b50038970 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:34.507 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.849+0000 7f7b6caac700 1 --2- 172.21.15.59:0/2204467813 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b68101400 0x7f7b68076a30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:34.507 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.849+0000 7f7b6caac700 1 -- 172.21.15.59:0/2204467813 >> 172.21.15.59:0/2204467813 conn(0x7f7b680fcae0 msgr2=0x7f7b680fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:34.507 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.849+0000 7f7b6caac700 1 -- 172.21.15.59:0/2204467813 shutdown_connections 2024-01-23T08:43:34.507 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:33.849+0000 7f7b6caac700 1 -- 172.21.15.59:0/2204467813 wait complete. 2024-01-23T08:43:34.806 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:34 smithi059 ceph-mon[28424]: mgrmap e8: smithi059.bzqirf(active, since 2s) 2024-01-23T08:43:34.806 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:34 smithi059 ceph-mon[28424]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:43:34.807 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:34 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' 2024-01-23T08:43:34.807 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:34 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:43:35.726 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout value unchanged 2024-01-23T08:43:35.726 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.981+0000 7fe31355b700 1 Processor -- start 2024-01-23T08:43:35.726 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.981+0000 7fe31355b700 1 -- start start 2024-01-23T08:43:35.727 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.981+0000 7fe31355b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe30c102450 0x7fe30c102860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:35.727 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.981+0000 7fe31355b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30c102da0 con 0x7fe30c102450 2024-01-23T08:43:35.727 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.981+0000 7fe3112f7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe30c102450 0x7fe30c102860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:35.727 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.981+0000 7fe3112f7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe30c102450 0x7fe30c102860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49402/0 (socket says 172.21.15.59:49402) 2024-01-23T08:43:35.727 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.981+0000 7fe3112f7700 1 -- 172.21.15.59:0/3477738007 learned_addr learned my addr 172.21.15.59:0/3477738007 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:35.727 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.982+0000 7fe3112f7700 1 -- 172.21.15.59:0/3477738007 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30c102ee0 con 0x7fe30c102450 2024-01-23T08:43:35.727 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.982+0000 7fe3112f7700 1 --2- 172.21.15.59:0/3477738007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe30c102450 0x7fe30c102860 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe3080098e0 tx=0x7fe308009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c9593e2cded9d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:35.727 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.982+0000 7fe303fff700 1 -- 172.21.15.59:0/3477738007 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe30800b860 con 0x7fe30c102450 2024-01-23T08:43:35.727 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.982+0000 7fe303fff700 1 -- 172.21.15.59:0/3477738007 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe30800be30 con 0x7fe30c102450 2024-01-23T08:43:35.727 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.982+0000 7fe303fff700 1 -- 172.21.15.59:0/3477738007 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe308010380 con 0x7fe30c102450 2024-01-23T08:43:35.727 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.982+0000 7fe31355b700 1 -- 172.21.15.59:0/3477738007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe30c102450 msgr2=0x7fe30c102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:35.727 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.982+0000 7fe31355b700 1 --2- 172.21.15.59:0/3477738007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe30c102450 0x7fe30c102860 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe3080098e0 tx=0x7fe308009250 comp rx=0 tx=0).stop 2024-01-23T08:43:35.727 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.982+0000 7fe31355b700 1 -- 172.21.15.59:0/3477738007 shutdown_connections 2024-01-23T08:43:35.727 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.982+0000 7fe31355b700 1 --2- 172.21.15.59:0/3477738007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe30c102450 0x7fe30c102860 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:35.727 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.982+0000 7fe31355b700 1 -- 172.21.15.59:0/3477738007 >> 172.21.15.59:0/3477738007 conn(0x7fe30c0fdbd0 msgr2=0x7fe30c0fffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:35.728 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.983+0000 7fe31355b700 1 -- 172.21.15.59:0/3477738007 shutdown_connections 2024-01-23T08:43:35.728 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.983+0000 7fe31355b700 1 -- 172.21.15.59:0/3477738007 wait complete. 2024-01-23T08:43:35.728 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.983+0000 7fe31355b700 1 Processor -- start 2024-01-23T08:43:35.728 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.983+0000 7fe31355b700 1 -- start start 2024-01-23T08:43:35.728 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.983+0000 7fe31355b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe30c102450 0x7fe30c078b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:35.728 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.983+0000 7fe31355b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30c079090 con 0x7fe30c102450 2024-01-23T08:43:35.728 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.983+0000 7fe3112f7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe30c102450 0x7fe30c078b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:35.728 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.983+0000 7fe3112f7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe30c102450 0x7fe30c078b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49414/0 (socket says 172.21.15.59:49414) 2024-01-23T08:43:35.728 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.983+0000 7fe3112f7700 1 -- 172.21.15.59:0/2672124954 learned_addr learned my addr 172.21.15.59:0/2672124954 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:35.728 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.984+0000 7fe3112f7700 1 -- 172.21.15.59:0/2672124954 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3080094f0 con 0x7fe30c102450 2024-01-23T08:43:35.728 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.984+0000 7fe3112f7700 1 --2- 172.21.15.59:0/2672124954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe30c102450 0x7fe30c078b50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe308006b00 tx=0x7fe308009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:35.728 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.984+0000 7fe301ffb700 1 -- 172.21.15.59:0/2672124954 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3080109b0 con 0x7fe30c102450 2024-01-23T08:43:35.728 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.984+0000 7fe301ffb700 1 -- 172.21.15.59:0/2672124954 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe30801a430 con 0x7fe30c102450 2024-01-23T08:43:35.728 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.984+0000 7fe31355b700 1 -- 172.21.15.59:0/2672124954 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe30c0771b0 con 0x7fe30c102450 2024-01-23T08:43:35.729 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.984+0000 7fe301ffb700 1 -- 172.21.15.59:0/2672124954 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3080245b0 con 0x7fe30c102450 2024-01-23T08:43:35.729 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.984+0000 7fe31355b700 1 -- 172.21.15.59:0/2672124954 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe30c077620 con 0x7fe30c102450 2024-01-23T08:43:35.729 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.985+0000 7fe301ffb700 1 -- 172.21.15.59:0/2672124954 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fe308024830 con 0x7fe30c102450 2024-01-23T08:43:35.729 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.985+0000 7fe301ffb700 1 --2- 172.21.15.59:0/2672124954 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fe2f80365f0 0x7fe2f8038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:35.729 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.985+0000 7fe31355b700 1 -- 172.21.15.59:0/2672124954 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe30c1042d0 con 0x7fe30c102450 2024-01-23T08:43:35.729 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.985+0000 7fe301ffb700 1 -- 172.21.15.59:0/2672124954 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe30804ae50 con 0x7fe30c102450 2024-01-23T08:43:35.729 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.985+0000 7fe310af6700 1 --2- 172.21.15.59:0/2672124954 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fe2f80365f0 0x7fe2f8038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:35.729 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.986+0000 7fe310af6700 1 --2- 172.21.15.59:0/2672124954 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fe2f80365f0 0x7fe2f8038aa0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc009990 tx=0x7fe2fc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:35.729 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:34.990+0000 7fe301ffb700 1 -- 172.21.15.59:0/2672124954 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe30802cb80 con 0x7fe30c102450 2024-01-23T08:43:35.729 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:35.184+0000 7fe31355b700 1 -- 172.21.15.59:0/2672124954 --> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fe30c077f70 con 0x7fe2f80365f0 2024-01-23T08:43:35.729 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:35.185+0000 7fe301ffb700 1 -- 172.21.15.59:0/2672124954 <== mgr.14120 v2:172.21.15.59:6800/1337467262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fe30c077f70 con 0x7fe2f80365f0 2024-01-23T08:43:35.729 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:35.186+0000 7fe31355b700 1 -- 172.21.15.59:0/2672124954 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fe2f80365f0 msgr2=0x7fe2f8038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:35.729 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:35.186+0000 7fe31355b700 1 --2- 172.21.15.59:0/2672124954 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fe2f80365f0 0x7fe2f8038aa0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc009990 tx=0x7fe2fc006e40 comp rx=0 tx=0).stop 2024-01-23T08:43:35.729 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:35.186+0000 7fe31355b700 1 -- 172.21.15.59:0/2672124954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe30c102450 msgr2=0x7fe30c078b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:35.729 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:35.186+0000 7fe31355b700 1 --2- 172.21.15.59:0/2672124954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe30c102450 0x7fe30c078b50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe308006b00 tx=0x7fe308009250 comp rx=0 tx=0).stop 2024-01-23T08:43:35.729 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:35.187+0000 7fe31355b700 1 -- 172.21.15.59:0/2672124954 shutdown_connections 2024-01-23T08:43:35.730 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:35.187+0000 7fe31355b700 1 --2- 172.21.15.59:0/2672124954 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fe2f80365f0 0x7fe2f8038aa0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:35.730 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:35.187+0000 7fe31355b700 1 --2- 172.21.15.59:0/2672124954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe30c102450 0x7fe30c078b50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:35.730 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:35.187+0000 7fe31355b700 1 -- 172.21.15.59:0/2672124954 >> 172.21.15.59:0/2672124954 conn(0x7fe30c0fdbd0 msgr2=0x7fe30c0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:35.730 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:35.187+0000 7fe31355b700 1 -- 172.21.15.59:0/2672124954 shutdown_connections 2024-01-23T08:43:35.730 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:35.187+0000 7fe31355b700 1 -- 172.21.15.59:0/2672124954 wait complete. 2024-01-23T08:43:35.730 INFO:teuthology.orchestra.run.smithi059.stdout:Generating ssh key... 2024-01-23T08:43:35.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:35 smithi059 ceph-mon[28424]: [23/Jan/2024:08:43:34] ENGINE Bus STARTING 2024-01-23T08:43:35.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:35 smithi059 ceph-mon[28424]: [23/Jan/2024:08:43:34] ENGINE Serving on https://172.21.15.59:7150 2024-01-23T08:43:35.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:35 smithi059 ceph-mon[28424]: [23/Jan/2024:08:43:34] ENGINE Bus STARTED 2024-01-23T08:43:35.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:35 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' 2024-01-23T08:43:35.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:35 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:43:37.216 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:36 smithi059 ceph-mon[28424]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:43:37.491 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.131+0000 7f24eb6b7700 1 Processor -- start 2024-01-23T08:43:37.491 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.131+0000 7f24eb6b7700 1 -- start start 2024-01-23T08:43:37.491 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.131+0000 7f24eb6b7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24e4101410 0x7f24e4101820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:37.491 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.131+0000 7f24eb6b7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e4101df0 con 0x7f24e4101410 2024-01-23T08:43:37.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.131+0000 7f24e9453700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24e4101410 0x7f24e4101820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:37.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.131+0000 7f24e9453700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24e4101410 0x7f24e4101820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49422/0 (socket says 172.21.15.59:49422) 2024-01-23T08:43:37.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.131+0000 7f24e9453700 1 -- 172.21.15.59:0/2242842586 learned_addr learned my addr 172.21.15.59:0/2242842586 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:37.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.132+0000 7f24e9453700 1 -- 172.21.15.59:0/2242842586 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e4102620 con 0x7f24e4101410 2024-01-23T08:43:37.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.132+0000 7f24e9453700 1 --2- 172.21.15.59:0/2242842586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24e4101410 0x7f24e4101820 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f24e000bb80 tx=0x7f24e000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=613cadc6a8787add server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:37.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.132+0000 7f24dbfff700 1 -- 172.21.15.59:0/2242842586 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24e000dd00 con 0x7f24e4101410 2024-01-23T08:43:37.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.132+0000 7f24dbfff700 1 -- 172.21.15.59:0/2242842586 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f24e000de60 con 0x7f24e4101410 2024-01-23T08:43:37.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.132+0000 7f24dbfff700 1 -- 172.21.15.59:0/2242842586 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24e0018840 con 0x7f24e4101410 2024-01-23T08:43:37.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.132+0000 7f24eb6b7700 1 -- 172.21.15.59:0/2242842586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24e4101410 msgr2=0x7f24e4101820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:37.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.132+0000 7f24eb6b7700 1 --2- 172.21.15.59:0/2242842586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24e4101410 0x7f24e4101820 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f24e000bb80 tx=0x7f24e000b4f0 comp rx=0 tx=0).stop 2024-01-23T08:43:37.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.132+0000 7f24eb6b7700 1 -- 172.21.15.59:0/2242842586 shutdown_connections 2024-01-23T08:43:37.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.132+0000 7f24eb6b7700 1 --2- 172.21.15.59:0/2242842586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24e4101410 0x7f24e4101820 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:37.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.132+0000 7f24eb6b7700 1 -- 172.21.15.59:0/2242842586 >> 172.21.15.59:0/2242842586 conn(0x7f24e40fcad0 msgr2=0x7f24e40fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:37.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.133+0000 7f24eb6b7700 1 -- 172.21.15.59:0/2242842586 shutdown_connections 2024-01-23T08:43:37.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.133+0000 7f24eb6b7700 1 -- 172.21.15.59:0/2242842586 wait complete. 2024-01-23T08:43:37.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.133+0000 7f24eb6b7700 1 Processor -- start 2024-01-23T08:43:37.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.133+0000 7f24eb6b7700 1 -- start start 2024-01-23T08:43:37.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.133+0000 7f24eb6b7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24e4101410 0x7f24e410efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:37.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.133+0000 7f24eb6b7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e410f510 con 0x7f24e4101410 2024-01-23T08:43:37.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.133+0000 7f24e9453700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24e4101410 0x7f24e410efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:37.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.133+0000 7f24e9453700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24e4101410 0x7f24e410efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49438/0 (socket says 172.21.15.59:49438) 2024-01-23T08:43:37.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.133+0000 7f24e9453700 1 -- 172.21.15.59:0/1107270448 learned_addr learned my addr 172.21.15.59:0/1107270448 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:37.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.134+0000 7f24e9453700 1 -- 172.21.15.59:0/1107270448 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e000b790 con 0x7f24e4101410 2024-01-23T08:43:37.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.134+0000 7f24e9453700 1 --2- 172.21.15.59:0/1107270448 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24e4101410 0x7f24e410efd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f24e0018460 tx=0x7f24e000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:37.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.134+0000 7f24d9ffb700 1 -- 172.21.15.59:0/1107270448 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24e00108e0 con 0x7f24e4101410 2024-01-23T08:43:37.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.134+0000 7f24d9ffb700 1 -- 172.21.15.59:0/1107270448 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f24e001d050 con 0x7f24e4101410 2024-01-23T08:43:37.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.134+0000 7f24d9ffb700 1 -- 172.21.15.59:0/1107270448 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24e00223d0 con 0x7f24e4101410 2024-01-23T08:43:37.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.134+0000 7f24eb6b7700 1 -- 172.21.15.59:0/1107270448 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24e410f710 con 0x7f24e4101410 2024-01-23T08:43:37.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.134+0000 7f24eb6b7700 1 -- 172.21.15.59:0/1107270448 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24e410fb20 con 0x7f24e4101410 2024-01-23T08:43:37.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.135+0000 7f24d9ffb700 1 -- 172.21.15.59:0/1107270448 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f24e0023470 con 0x7f24e4101410 2024-01-23T08:43:37.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.135+0000 7f24d9ffb700 1 --2- 172.21.15.59:0/1107270448 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f24d0036640 0x7f24d0038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:37.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.135+0000 7f24eb6b7700 1 -- 172.21.15.59:0/1107270448 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24e4108f60 con 0x7f24e4101410 2024-01-23T08:43:37.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.135+0000 7f24d9ffb700 1 -- 172.21.15.59:0/1107270448 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f24e0014070 con 0x7f24e4101410 2024-01-23T08:43:37.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.135+0000 7f24e8c52700 1 --2- 172.21.15.59:0/1107270448 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f24d0036640 0x7f24d0038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:37.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.136+0000 7f24e8c52700 1 --2- 172.21.15.59:0/1107270448 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f24d0036640 0x7f24d0038af0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f24d4009990 tx=0x7f24d4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:37.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.140+0000 7f24d9ffb700 1 -- 172.21.15.59:0/1107270448 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f24e0010a40 con 0x7f24e4101410 2024-01-23T08:43:37.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.330+0000 7f24eb6b7700 1 -- 172.21.15.59:0/1107270448 --> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f24e405e260 con 0x7f24d0036640 2024-01-23T08:43:37.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.930+0000 7f24d9ffb700 1 -- 172.21.15.59:0/1107270448 <== mgr.14120 v2:172.21.15.59:6800/1337467262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f24e405e260 con 0x7f24d0036640 2024-01-23T08:43:37.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.932+0000 7f24eb6b7700 1 -- 172.21.15.59:0/1107270448 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f24d0036640 msgr2=0x7f24d0038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:37.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.932+0000 7f24eb6b7700 1 --2- 172.21.15.59:0/1107270448 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f24d0036640 0x7f24d0038af0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f24d4009990 tx=0x7f24d4006e40 comp rx=0 tx=0).stop 2024-01-23T08:43:37.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.932+0000 7f24eb6b7700 1 -- 172.21.15.59:0/1107270448 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24e4101410 msgr2=0x7f24e410efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:37.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.932+0000 7f24eb6b7700 1 --2- 172.21.15.59:0/1107270448 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24e4101410 0x7f24e410efd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f24e0018460 tx=0x7f24e000b4f0 comp rx=0 tx=0).stop 2024-01-23T08:43:37.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.932+0000 7f24eb6b7700 1 -- 172.21.15.59:0/1107270448 shutdown_connections 2024-01-23T08:43:37.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.932+0000 7f24eb6b7700 1 --2- 172.21.15.59:0/1107270448 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f24d0036640 0x7f24d0038af0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:37.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.932+0000 7f24eb6b7700 1 --2- 172.21.15.59:0/1107270448 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f24e4101410 0x7f24e410efd0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:37.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.932+0000 7f24eb6b7700 1 -- 172.21.15.59:0/1107270448 >> 172.21.15.59:0/1107270448 conn(0x7f24e40fcad0 msgr2=0x7f24e41060e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:37.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.932+0000 7f24eb6b7700 1 -- 172.21.15.59:0/1107270448 shutdown_connections 2024-01-23T08:43:37.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:36.932+0000 7f24eb6b7700 1 -- 172.21.15.59:0/1107270448 wait complete. 2024-01-23T08:43:38.106 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:37 smithi059 ceph-mon[28424]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:43:38.106 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:37 smithi059 ceph-mon[28424]: Generating ssh key... 2024-01-23T08:43:38.107 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:37 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' 2024-01-23T08:43:38.107 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:37 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' 2024-01-23T08:43:38.654 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCZRpFiSdBf0oMP09S9Y1pzIjzVFOtgF6EM2wE+dTtB6bGYhBlz9lrEvoVQdFAdhlPYVoV2uiIQGkmXAuPvp+sAqqjjZ+842+L0TKRakwwYXZaKVkiJ174NUWhTlAviPc2BAAeTRZTrsXi8Duwd/jdfegi/Nu9GQl2FO13ARWXUUHJPxX8zUe81Tv0rOGjZyQe/2iTsXrdWpHeBIk0PLvnw3Vaq0ttKruOQkRs4ikc1uOl1e3iIeVrnC/DEDS/xl2Vm6adh2phaMh2Wo9ngmkNzLLjNbiXU4BePS8G8RrF3cgdBojbW8fRovi2RLgEA1WKZ6hOthH3b90AkKnyx9k6DcS6Y3jsdPMxL3x3gJV9fXRlCPLPy+CCrC1aY2udIHE6y1DSMRCpdg3oAj2C3M4Vy7CVZ5fMIygt2hqYwpw1MMCUui5kQbW+DeTuy3hdSDA9MSB1Ok+FbW1kCLlJmIanzxtG33J651IJuDwHswF0PFilNy++3tzntKCL33rJ/wnc= ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715 2024-01-23T08:43:38.655 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.905+0000 7fc5cda57700 1 Processor -- start 2024-01-23T08:43:38.655 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.905+0000 7fc5cda57700 1 -- start start 2024-01-23T08:43:38.655 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.905+0000 7fc5cda57700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5c8102500 0x7fc5c8102910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:38.655 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.905+0000 7fc5cda57700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c8102e50 con 0x7fc5c8102500 2024-01-23T08:43:38.655 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.905+0000 7fc5c6ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5c8102500 0x7fc5c8102910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:38.655 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.905+0000 7fc5c6ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5c8102500 0x7fc5c8102910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37052/0 (socket says 172.21.15.59:37052) 2024-01-23T08:43:38.655 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.905+0000 7fc5c6ffd700 1 -- 172.21.15.59:0/2344787858 learned_addr learned my addr 172.21.15.59:0/2344787858 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:38.655 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.906+0000 7fc5c6ffd700 1 -- 172.21.15.59:0/2344787858 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c8102f90 con 0x7fc5c8102500 2024-01-23T08:43:38.655 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.906+0000 7fc5c6ffd700 1 --2- 172.21.15.59:0/2344787858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5c8102500 0x7fc5c8102910 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc5c00098e0 tx=0x7fc5c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28350d39451aaa04 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:38.655 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.906+0000 7fc5c5ffb700 1 -- 172.21.15.59:0/2344787858 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5c000b860 con 0x7fc5c8102500 2024-01-23T08:43:38.655 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.906+0000 7fc5c5ffb700 1 -- 172.21.15.59:0/2344787858 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fc5c000be30 con 0x7fc5c8102500 2024-01-23T08:43:38.655 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.906+0000 7fc5c5ffb700 1 -- 172.21.15.59:0/2344787858 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5c0010380 con 0x7fc5c8102500 2024-01-23T08:43:38.656 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.907+0000 7fc5cda57700 1 -- 172.21.15.59:0/2344787858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5c8102500 msgr2=0x7fc5c8102910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:38.656 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.907+0000 7fc5cda57700 1 --2- 172.21.15.59:0/2344787858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5c8102500 0x7fc5c8102910 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc5c00098e0 tx=0x7fc5c0009250 comp rx=0 tx=0).stop 2024-01-23T08:43:38.656 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.907+0000 7fc5cda57700 1 -- 172.21.15.59:0/2344787858 shutdown_connections 2024-01-23T08:43:38.656 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.907+0000 7fc5cda57700 1 --2- 172.21.15.59:0/2344787858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5c8102500 0x7fc5c8102910 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:38.656 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.907+0000 7fc5cda57700 1 -- 172.21.15.59:0/2344787858 >> 172.21.15.59:0/2344787858 conn(0x7fc5c80fdca0 msgr2=0x7fc5c81000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:38.656 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.907+0000 7fc5cda57700 1 -- 172.21.15.59:0/2344787858 shutdown_connections 2024-01-23T08:43:38.656 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.907+0000 7fc5cda57700 1 -- 172.21.15.59:0/2344787858 wait complete. 2024-01-23T08:43:38.656 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.907+0000 7fc5cda57700 1 Processor -- start 2024-01-23T08:43:38.656 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.907+0000 7fc5cda57700 1 -- start start 2024-01-23T08:43:38.656 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.908+0000 7fc5cda57700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5c8102500 0x7fc5c8111490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:38.656 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.908+0000 7fc5cda57700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c81119d0 con 0x7fc5c8102500 2024-01-23T08:43:38.656 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.908+0000 7fc5c6ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5c8102500 0x7fc5c8111490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:38.656 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.908+0000 7fc5c6ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5c8102500 0x7fc5c8111490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37068/0 (socket says 172.21.15.59:37068) 2024-01-23T08:43:38.657 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.908+0000 7fc5c6ffd700 1 -- 172.21.15.59:0/1409341668 learned_addr learned my addr 172.21.15.59:0/1409341668 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:38.657 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.908+0000 7fc5c6ffd700 1 -- 172.21.15.59:0/1409341668 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c00094f0 con 0x7fc5c8102500 2024-01-23T08:43:38.657 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.908+0000 7fc5c6ffd700 1 --2- 172.21.15.59:0/1409341668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5c8102500 0x7fc5c8111490 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0006b00 tx=0x7fc5c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:38.657 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.908+0000 7fc5bf7fe700 1 -- 172.21.15.59:0/1409341668 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5c00109b0 con 0x7fc5c8102500 2024-01-23T08:43:38.657 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.908+0000 7fc5bf7fe700 1 -- 172.21.15.59:0/1409341668 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fc5c001a430 con 0x7fc5c8102500 2024-01-23T08:43:38.657 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.908+0000 7fc5cda57700 1 -- 172.21.15.59:0/1409341668 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c8111bd0 con 0x7fc5c8102500 2024-01-23T08:43:38.657 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.908+0000 7fc5bf7fe700 1 -- 172.21.15.59:0/1409341668 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5c00245b0 con 0x7fc5c8102500 2024-01-23T08:43:38.657 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.908+0000 7fc5cda57700 1 -- 172.21.15.59:0/1409341668 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c8111fe0 con 0x7fc5c8102500 2024-01-23T08:43:38.657 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.909+0000 7fc5bf7fe700 1 -- 172.21.15.59:0/1409341668 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fc5c00247e0 con 0x7fc5c8102500 2024-01-23T08:43:38.657 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.909+0000 7fc5cda57700 1 -- 172.21.15.59:0/1409341668 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c810b380 con 0x7fc5c8102500 2024-01-23T08:43:38.657 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.909+0000 7fc5bf7fe700 1 --2- 172.21.15.59:0/1409341668 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc5b4036640 0x7fc5b4038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:38.657 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.909+0000 7fc5bf7fe700 1 -- 172.21.15.59:0/1409341668 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fc5c004aba0 con 0x7fc5c8102500 2024-01-23T08:43:38.657 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.910+0000 7fc5c67fc700 1 --2- 172.21.15.59:0/1409341668 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc5b4036640 0x7fc5b4038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:38.657 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.910+0000 7fc5c67fc700 1 --2- 172.21.15.59:0/1409341668 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc5b4036640 0x7fc5b4038af0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009990 tx=0x7fc5b0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:38.658 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:37.914+0000 7fc5bf7fe700 1 -- 172.21.15.59:0/1409341668 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc5c0018350 con 0x7fc5c8102500 2024-01-23T08:43:38.658 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:38.109+0000 7fc5cda57700 1 -- 172.21.15.59:0/1409341668 --> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5c805e260 con 0x7fc5b4036640 2024-01-23T08:43:38.658 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:38.109+0000 7fc5bf7fe700 1 -- 172.21.15.59:0/1409341668 <== mgr.14120 v2:172.21.15.59:6800/1337467262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fc5c805e260 con 0x7fc5b4036640 2024-01-23T08:43:38.658 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:38.111+0000 7fc5cda57700 1 -- 172.21.15.59:0/1409341668 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc5b4036640 msgr2=0x7fc5b4038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:38.658 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:38.111+0000 7fc5cda57700 1 --2- 172.21.15.59:0/1409341668 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc5b4036640 0x7fc5b4038af0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009990 tx=0x7fc5b0006e40 comp rx=0 tx=0).stop 2024-01-23T08:43:38.658 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:38.111+0000 7fc5cda57700 1 -- 172.21.15.59:0/1409341668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5c8102500 msgr2=0x7fc5c8111490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:38.658 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:38.111+0000 7fc5cda57700 1 --2- 172.21.15.59:0/1409341668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5c8102500 0x7fc5c8111490 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0006b00 tx=0x7fc5c0009250 comp rx=0 tx=0).stop 2024-01-23T08:43:38.658 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:38.111+0000 7fc5cda57700 1 -- 172.21.15.59:0/1409341668 shutdown_connections 2024-01-23T08:43:38.658 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:38.111+0000 7fc5cda57700 1 --2- 172.21.15.59:0/1409341668 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc5b4036640 0x7fc5b4038af0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:38.658 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:38.111+0000 7fc5cda57700 1 --2- 172.21.15.59:0/1409341668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5c8102500 0x7fc5c8111490 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:38.658 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:38.111+0000 7fc5cda57700 1 -- 172.21.15.59:0/1409341668 >> 172.21.15.59:0/1409341668 conn(0x7fc5c80fdca0 msgr2=0x7fc5c80feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:38.658 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:38.112+0000 7fc5cda57700 1 -- 172.21.15.59:0/1409341668 shutdown_connections 2024-01-23T08:43:38.658 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:38.112+0000 7fc5cda57700 1 -- 172.21.15.59:0/1409341668 wait complete. 2024-01-23T08:43:38.658 INFO:teuthology.orchestra.run.smithi059.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-01-23T08:43:38.659 INFO:teuthology.orchestra.run.smithi059.stdout:Adding key to root@localhost authorized_keys... 2024-01-23T08:43:38.659 INFO:teuthology.orchestra.run.smithi059.stdout:Adding host smithi059... 2024-01-23T08:43:39.305 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:39 smithi059 ceph-mon[28424]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:43:41.223 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:40 smithi059 ceph-mon[28424]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi059", "addr": "172.21.15.59", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:43:41.223 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:40 smithi059 ceph-mon[28424]: Deploying cephadm binary to smithi059 2024-01-23T08:43:41.898 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Added host 'smithi059' with addr '172.21.15.59' 2024-01-23T08:43:41.898 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.027+0000 7f2e01f2c700 1 Processor -- start 2024-01-23T08:43:41.898 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.027+0000 7f2e01f2c700 1 -- start start 2024-01-23T08:43:41.898 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.027+0000 7f2e01f2c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc102450 0x7f2dfc102860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:41.898 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.027+0000 7f2e01f2c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc102da0 con 0x7f2dfc102450 2024-01-23T08:43:41.898 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.027+0000 7f2dfb7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc102450 0x7f2dfc102860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:41.898 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.027+0000 7f2dfb7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc102450 0x7f2dfc102860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37084/0 (socket says 172.21.15.59:37084) 2024-01-23T08:43:41.898 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.027+0000 7f2dfb7fe700 1 -- 172.21.15.59:0/2505604856 learned_addr learned my addr 172.21.15.59:0/2505604856 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:41.899 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.027+0000 7f2dfb7fe700 1 -- 172.21.15.59:0/2505604856 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dfc102ee0 con 0x7f2dfc102450 2024-01-23T08:43:41.899 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.028+0000 7f2dfb7fe700 1 --2- 172.21.15.59:0/2505604856 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc102450 0x7f2dfc102860 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2dec0098e0 tx=0x7f2dec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a9d3cfea7f3d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:41.899 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.028+0000 7f2dfa7fc700 1 -- 172.21.15.59:0/2505604856 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dec00b860 con 0x7f2dfc102450 2024-01-23T08:43:41.899 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.028+0000 7f2dfa7fc700 1 -- 172.21.15.59:0/2505604856 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f2dec00be30 con 0x7f2dfc102450 2024-01-23T08:43:41.899 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.028+0000 7f2dfa7fc700 1 -- 172.21.15.59:0/2505604856 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dec010380 con 0x7f2dfc102450 2024-01-23T08:43:41.899 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.028+0000 7f2e01f2c700 1 -- 172.21.15.59:0/2505604856 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc102450 msgr2=0x7f2dfc102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:41.899 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.028+0000 7f2e01f2c700 1 --2- 172.21.15.59:0/2505604856 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc102450 0x7f2dfc102860 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2dec0098e0 tx=0x7f2dec009250 comp rx=0 tx=0).stop 2024-01-23T08:43:41.899 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.028+0000 7f2e01f2c700 1 -- 172.21.15.59:0/2505604856 shutdown_connections 2024-01-23T08:43:41.899 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.028+0000 7f2e01f2c700 1 --2- 172.21.15.59:0/2505604856 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc102450 0x7f2dfc102860 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:41.899 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.028+0000 7f2e01f2c700 1 -- 172.21.15.59:0/2505604856 >> 172.21.15.59:0/2505604856 conn(0x7f2dfc0fdbd0 msgr2=0x7f2dfc0fffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:41.899 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.028+0000 7f2e01f2c700 1 -- 172.21.15.59:0/2505604856 shutdown_connections 2024-01-23T08:43:41.899 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.029+0000 7f2e01f2c700 1 -- 172.21.15.59:0/2505604856 wait complete. 2024-01-23T08:43:41.899 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.029+0000 7f2e01f2c700 1 Processor -- start 2024-01-23T08:43:41.899 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.029+0000 7f2e01f2c700 1 -- start start 2024-01-23T08:43:41.899 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.029+0000 7f2e01f2c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc102450 0x7f2dfc078b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:41.900 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.029+0000 7f2e01f2c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc079090 con 0x7f2dfc102450 2024-01-23T08:43:41.900 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.030+0000 7f2dfb7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc102450 0x7f2dfc078b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:41.900 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.030+0000 7f2dfb7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc102450 0x7f2dfc078b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37094/0 (socket says 172.21.15.59:37094) 2024-01-23T08:43:41.900 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.030+0000 7f2dfb7fe700 1 -- 172.21.15.59:0/2672551709 learned_addr learned my addr 172.21.15.59:0/2672551709 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:41.900 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.030+0000 7f2dfb7fe700 1 -- 172.21.15.59:0/2672551709 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dec0094f0 con 0x7f2dfc102450 2024-01-23T08:43:41.900 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.030+0000 7f2dfb7fe700 1 --2- 172.21.15.59:0/2672551709 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc102450 0x7f2dfc078b50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2dec000c00 tx=0x7f2dec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:41.900 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.030+0000 7f2de3fff700 1 -- 172.21.15.59:0/2672551709 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dec010a80 con 0x7f2dfc102450 2024-01-23T08:43:41.900 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.030+0000 7f2e01f2c700 1 -- 172.21.15.59:0/2672551709 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dfc0771b0 con 0x7f2dfc102450 2024-01-23T08:43:41.900 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.030+0000 7f2de3fff700 1 -- 172.21.15.59:0/2672551709 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f2dec01a430 con 0x7f2dfc102450 2024-01-23T08:43:41.900 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.030+0000 7f2de3fff700 1 -- 172.21.15.59:0/2672551709 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dec0245b0 con 0x7f2dfc102450 2024-01-23T08:43:41.900 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.030+0000 7f2e01f2c700 1 -- 172.21.15.59:0/2672551709 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dfc077620 con 0x7f2dfc102450 2024-01-23T08:43:41.900 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.031+0000 7f2de3fff700 1 -- 172.21.15.59:0/2672551709 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f2dec024870 con 0x7f2dfc102450 2024-01-23T08:43:41.900 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.031+0000 7f2e01f2c700 1 -- 172.21.15.59:0/2672551709 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dfc05f450 con 0x7f2dfc102450 2024-01-23T08:43:41.900 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.031+0000 7f2de3fff700 1 --2- 172.21.15.59:0/2672551709 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f2de40365f0 0x7f2de4038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:41.901 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.032+0000 7f2de3fff700 1 -- 172.21.15.59:0/2672551709 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f2dec04a890 con 0x7f2dfc102450 2024-01-23T08:43:41.901 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.032+0000 7f2dfaffd700 1 --2- 172.21.15.59:0/2672551709 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f2de40365f0 0x7f2de4038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:41.901 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.032+0000 7f2dfaffd700 1 --2- 172.21.15.59:0/2672551709 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f2de40365f0 0x7f2de4038aa0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2df0009990 tx=0x7f2df0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:41.901 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.036+0000 7f2de3fff700 1 -- 172.21.15.59:0/2672551709 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2dec02c340 con 0x7f2dfc102450 2024-01-23T08:43:41.901 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:39.234+0000 7f2e01f2c700 1 -- 172.21.15.59:0/2672551709 --> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi059", "addr": "172.21.15.59", "target": ["mon-mgr", ""]}) v1 -- 0x7f2dfc077de0 con 0x7f2de40365f0 2024-01-23T08:43:41.901 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:41.319+0000 7f2de3fff700 1 -- 172.21.15.59:0/2672551709 <== mgr.14120 v2:172.21.15.59:6800/1337467262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f2dfc077de0 con 0x7f2de40365f0 2024-01-23T08:43:41.901 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:41.321+0000 7f2e01f2c700 1 -- 172.21.15.59:0/2672551709 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f2de40365f0 msgr2=0x7f2de4038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:41.901 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:41.321+0000 7f2e01f2c700 1 --2- 172.21.15.59:0/2672551709 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f2de40365f0 0x7f2de4038aa0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2df0009990 tx=0x7f2df0006e40 comp rx=0 tx=0).stop 2024-01-23T08:43:41.901 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:41.322+0000 7f2e01f2c700 1 -- 172.21.15.59:0/2672551709 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc102450 msgr2=0x7f2dfc078b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:41.901 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:41.322+0000 7f2e01f2c700 1 --2- 172.21.15.59:0/2672551709 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc102450 0x7f2dfc078b50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2dec000c00 tx=0x7f2dec009250 comp rx=0 tx=0).stop 2024-01-23T08:43:41.901 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:41.322+0000 7f2e01f2c700 1 -- 172.21.15.59:0/2672551709 shutdown_connections 2024-01-23T08:43:41.901 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:41.322+0000 7f2e01f2c700 1 --2- 172.21.15.59:0/2672551709 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f2de40365f0 0x7f2de4038aa0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:41.901 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:41.322+0000 7f2e01f2c700 1 --2- 172.21.15.59:0/2672551709 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dfc102450 0x7f2dfc078b50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:41.902 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:41.322+0000 7f2e01f2c700 1 -- 172.21.15.59:0/2672551709 >> 172.21.15.59:0/2672551709 conn(0x7f2dfc0fdbd0 msgr2=0x7f2dfc0ff2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:41.902 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:41.322+0000 7f2e01f2c700 1 -- 172.21.15.59:0/2672551709 shutdown_connections 2024-01-23T08:43:41.902 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:41.322+0000 7f2e01f2c700 1 -- 172.21.15.59:0/2672551709 wait complete. 2024-01-23T08:43:41.902 INFO:teuthology.orchestra.run.smithi059.stdout:Deploying mon service with default placement... 2024-01-23T08:43:42.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:42 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' 2024-01-23T08:43:42.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:42 smithi059 ceph-mon[28424]: Added host smithi059 2024-01-23T08:43:42.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:42 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:43:43.275 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-01-23T08:43:43.275 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.357+0000 7f5fb146f700 1 Processor -- start 2024-01-23T08:43:43.275 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.357+0000 7f5fb146f700 1 -- start start 2024-01-23T08:43:43.275 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.357+0000 7f5fb146f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fac1029e0 0x7f5fac102df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:43.275 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.358+0000 7f5fb146f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fac1033c0 con 0x7f5fac1029e0 2024-01-23T08:43:43.275 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.358+0000 7f5faaffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fac1029e0 0x7f5fac102df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:43.275 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.358+0000 7f5faaffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fac1029e0 0x7f5fac102df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37104/0 (socket says 172.21.15.59:37104) 2024-01-23T08:43:43.275 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.358+0000 7f5faaffd700 1 -- 172.21.15.59:0/2167047565 learned_addr learned my addr 172.21.15.59:0/2167047565 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:43.276 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.358+0000 7f5faaffd700 1 -- 172.21.15.59:0/2167047565 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fac103500 con 0x7f5fac1029e0 2024-01-23T08:43:43.276 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.358+0000 7f5faaffd700 1 --2- 172.21.15.59:0/2167047565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fac1029e0 0x7f5fac102df0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c0098e0 tx=0x7f5f9c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cb540766c63a1a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:43.276 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.358+0000 7f5fa9ffb700 1 -- 172.21.15.59:0/2167047565 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f9c00b860 con 0x7f5fac1029e0 2024-01-23T08:43:43.276 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.358+0000 7f5fa9ffb700 1 -- 172.21.15.59:0/2167047565 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f5f9c00be30 con 0x7f5fac1029e0 2024-01-23T08:43:43.276 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.358+0000 7f5fa9ffb700 1 -- 172.21.15.59:0/2167047565 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f9c010380 con 0x7f5fac1029e0 2024-01-23T08:43:43.276 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.359+0000 7f5fb146f700 1 -- 172.21.15.59:0/2167047565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fac1029e0 msgr2=0x7f5fac102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:43.276 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.359+0000 7f5fb146f700 1 --2- 172.21.15.59:0/2167047565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fac1029e0 0x7f5fac102df0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c0098e0 tx=0x7f5f9c009250 comp rx=0 tx=0).stop 2024-01-23T08:43:43.276 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.359+0000 7f5fb146f700 1 -- 172.21.15.59:0/2167047565 shutdown_connections 2024-01-23T08:43:43.276 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.359+0000 7f5fb146f700 1 --2- 172.21.15.59:0/2167047565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fac1029e0 0x7f5fac102df0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:43.276 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.359+0000 7f5fb146f700 1 -- 172.21.15.59:0/2167047565 >> 172.21.15.59:0/2167047565 conn(0x7f5fac0fc260 msgr2=0x7f5fac0fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:43.276 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.359+0000 7f5fb146f700 1 -- 172.21.15.59:0/2167047565 shutdown_connections 2024-01-23T08:43:43.276 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.359+0000 7f5fb146f700 1 -- 172.21.15.59:0/2167047565 wait complete. 2024-01-23T08:43:43.276 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.360+0000 7f5fb146f700 1 Processor -- start 2024-01-23T08:43:43.277 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.360+0000 7f5fb146f700 1 -- start start 2024-01-23T08:43:43.277 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.360+0000 7f5fb146f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fac1029e0 0x7f5fac1010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:43.277 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.360+0000 7f5fb146f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fac113fc0 con 0x7f5fac1029e0 2024-01-23T08:43:43.277 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.360+0000 7f5faaffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fac1029e0 0x7f5fac1010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:43.277 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.360+0000 7f5faaffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fac1029e0 0x7f5fac1010a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37112/0 (socket says 172.21.15.59:37112) 2024-01-23T08:43:43.277 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.360+0000 7f5faaffd700 1 -- 172.21.15.59:0/2688925836 learned_addr learned my addr 172.21.15.59:0/2688925836 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:43.277 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.360+0000 7f5faaffd700 1 -- 172.21.15.59:0/2688925836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9c0094f0 con 0x7f5fac1029e0 2024-01-23T08:43:43.277 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.360+0000 7f5faaffd700 1 --2- 172.21.15.59:0/2688925836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fac1029e0 0x7f5fac1010a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c006d40 tx=0x7f5f9c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:43.277 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.361+0000 7f5f937fe700 1 -- 172.21.15.59:0/2688925836 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f9c0109b0 con 0x7f5fac1029e0 2024-01-23T08:43:43.277 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.361+0000 7f5f937fe700 1 -- 172.21.15.59:0/2688925836 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f5f9c024440 con 0x7f5fac1029e0 2024-01-23T08:43:43.277 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.361+0000 7f5fb146f700 1 -- 172.21.15.59:0/2688925836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fac114100 con 0x7f5fac1029e0 2024-01-23T08:43:43.277 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.361+0000 7f5f937fe700 1 -- 172.21.15.59:0/2688925836 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f9c01a5f0 con 0x7f5fac1029e0 2024-01-23T08:43:43.277 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.361+0000 7f5fb146f700 1 -- 172.21.15.59:0/2688925836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fac114450 con 0x7f5fac1029e0 2024-01-23T08:43:43.277 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.361+0000 7f5f937fe700 1 -- 172.21.15.59:0/2688925836 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f5f9c01a820 con 0x7f5fac1029e0 2024-01-23T08:43:43.277 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.362+0000 7f5f937fe700 1 --2- 172.21.15.59:0/2688925836 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f5f94036640 0x7f5f94038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:43.278 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.362+0000 7f5f937fe700 1 -- 172.21.15.59:0/2688925836 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f5f9c04b3b0 con 0x7f5fac1029e0 2024-01-23T08:43:43.278 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.362+0000 7f5faa7fc700 1 --2- 172.21.15.59:0/2688925836 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f5f94036640 0x7f5f94038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:43.278 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.362+0000 7f5fb146f700 1 -- 172.21.15.59:0/2688925836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fac10a300 con 0x7f5fac1029e0 2024-01-23T08:43:43.278 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.362+0000 7f5faa7fc700 1 --2- 172.21.15.59:0/2688925836 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f5f94036640 0x7f5f94038af0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0009990 tx=0x7f5fa0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:43.278 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.367+0000 7f5f937fe700 1 -- 172.21.15.59:0/2688925836 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5f9c02d3f0 con 0x7f5fac1029e0 2024-01-23T08:43:43.278 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.602+0000 7f5fb146f700 1 -- 172.21.15.59:0/2688925836 --> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f5fac02b350 con 0x7f5f94036640 2024-01-23T08:43:43.278 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.610+0000 7f5f937fe700 1 -- 172.21.15.59:0/2688925836 <== mgr.14120 v2:172.21.15.59:6800/1337467262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f5fac02b350 con 0x7f5f94036640 2024-01-23T08:43:43.278 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.612+0000 7f5fb146f700 1 -- 172.21.15.59:0/2688925836 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f5f94036640 msgr2=0x7f5f94038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:43.278 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.612+0000 7f5fb146f700 1 --2- 172.21.15.59:0/2688925836 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f5f94036640 0x7f5f94038af0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0009990 tx=0x7f5fa0006e40 comp rx=0 tx=0).stop 2024-01-23T08:43:43.278 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.612+0000 7f5fb146f700 1 -- 172.21.15.59:0/2688925836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fac1029e0 msgr2=0x7f5fac1010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:43.278 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.612+0000 7f5fb146f700 1 --2- 172.21.15.59:0/2688925836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fac1029e0 0x7f5fac1010a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c006d40 tx=0x7f5f9c009250 comp rx=0 tx=0).stop 2024-01-23T08:43:43.278 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.612+0000 7f5fb146f700 1 -- 172.21.15.59:0/2688925836 shutdown_connections 2024-01-23T08:43:43.278 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.612+0000 7f5fb146f700 1 --2- 172.21.15.59:0/2688925836 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f5f94036640 0x7f5f94038af0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:43.278 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.612+0000 7f5fb146f700 1 --2- 172.21.15.59:0/2688925836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fac1029e0 0x7f5fac1010a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:43.279 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.612+0000 7f5fb146f700 1 -- 172.21.15.59:0/2688925836 >> 172.21.15.59:0/2688925836 conn(0x7f5fac0fc260 msgr2=0x7f5fac0fd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:43.279 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.612+0000 7f5fb146f700 1 -- 172.21.15.59:0/2688925836 shutdown_connections 2024-01-23T08:43:43.279 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:42.612+0000 7f5fb146f700 1 -- 172.21.15.59:0/2688925836 wait complete. 2024-01-23T08:43:43.279 INFO:teuthology.orchestra.run.smithi059.stdout:Deploying mgr service with default placement... 2024-01-23T08:43:43.866 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:43 smithi059 ceph-mon[28424]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:43:43.866 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:43 smithi059 ceph-mon[28424]: Saving service mon spec with placement count:5 2024-01-23T08:43:43.866 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:43 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' 2024-01-23T08:43:44.523 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-01-23T08:43:44.523 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.668+0000 7feca6509700 1 Processor -- start 2024-01-23T08:43:44.523 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.668+0000 7feca6509700 1 -- start start 2024-01-23T08:43:44.523 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.668+0000 7feca6509700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca0102480 0x7feca0102890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:44.523 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.668+0000 7feca6509700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca0102dd0 con 0x7feca0102480 2024-01-23T08:43:44.523 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.668+0000 7fec9ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca0102480 0x7feca0102890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:44.524 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.668+0000 7fec9ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca0102480 0x7feca0102890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37116/0 (socket says 172.21.15.59:37116) 2024-01-23T08:43:44.524 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.668+0000 7fec9ffff700 1 -- 172.21.15.59:0/3503759863 learned_addr learned my addr 172.21.15.59:0/3503759863 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:44.524 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.669+0000 7fec9ffff700 1 -- 172.21.15.59:0/3503759863 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca0102f10 con 0x7feca0102480 2024-01-23T08:43:44.524 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.669+0000 7fec9ffff700 1 --2- 172.21.15.59:0/3503759863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca0102480 0x7feca0102890 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fec900098e0 tx=0x7fec90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2104da73076a14b server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:44.524 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.669+0000 7fec9effd700 1 -- 172.21.15.59:0/3503759863 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec9000b860 con 0x7feca0102480 2024-01-23T08:43:44.524 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.669+0000 7fec9effd700 1 -- 172.21.15.59:0/3503759863 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fec9000be30 con 0x7feca0102480 2024-01-23T08:43:44.524 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.669+0000 7fec9effd700 1 -- 172.21.15.59:0/3503759863 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec90010380 con 0x7feca0102480 2024-01-23T08:43:44.524 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.670+0000 7feca6509700 1 -- 172.21.15.59:0/3503759863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca0102480 msgr2=0x7feca0102890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:44.524 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.670+0000 7feca6509700 1 --2- 172.21.15.59:0/3503759863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca0102480 0x7feca0102890 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fec900098e0 tx=0x7fec90009250 comp rx=0 tx=0).stop 2024-01-23T08:43:44.524 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.670+0000 7feca6509700 1 -- 172.21.15.59:0/3503759863 shutdown_connections 2024-01-23T08:43:44.524 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.670+0000 7feca6509700 1 --2- 172.21.15.59:0/3503759863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca0102480 0x7feca0102890 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:44.524 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.670+0000 7feca6509700 1 -- 172.21.15.59:0/3503759863 >> 172.21.15.59:0/3503759863 conn(0x7feca00fdbe0 msgr2=0x7feca0100010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:44.524 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.670+0000 7feca6509700 1 -- 172.21.15.59:0/3503759863 shutdown_connections 2024-01-23T08:43:44.525 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.670+0000 7feca6509700 1 -- 172.21.15.59:0/3503759863 wait complete. 2024-01-23T08:43:44.525 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.670+0000 7feca6509700 1 Processor -- start 2024-01-23T08:43:44.525 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.670+0000 7feca6509700 1 -- start start 2024-01-23T08:43:44.525 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.671+0000 7feca6509700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca0102480 0x7feca0111140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:44.525 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.671+0000 7feca6509700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca0111680 con 0x7feca0102480 2024-01-23T08:43:44.525 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.671+0000 7fec9ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca0102480 0x7feca0111140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:44.525 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.671+0000 7fec9ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca0102480 0x7feca0111140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37132/0 (socket says 172.21.15.59:37132) 2024-01-23T08:43:44.525 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.671+0000 7fec9ffff700 1 -- 172.21.15.59:0/1331627247 learned_addr learned my addr 172.21.15.59:0/1331627247 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:44.525 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.671+0000 7fec9ffff700 1 -- 172.21.15.59:0/1331627247 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec900094f0 con 0x7feca0102480 2024-01-23T08:43:44.525 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.671+0000 7fec9ffff700 1 --2- 172.21.15.59:0/1331627247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca0102480 0x7feca0111140 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fec90000c00 tx=0x7fec90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:44.525 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.671+0000 7fec9cff9700 1 -- 172.21.15.59:0/1331627247 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec90010a80 con 0x7feca0102480 2024-01-23T08:43:44.525 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.671+0000 7fec9cff9700 1 -- 172.21.15.59:0/1331627247 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fec9001a430 con 0x7feca0102480 2024-01-23T08:43:44.525 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.671+0000 7fec9cff9700 1 -- 172.21.15.59:0/1331627247 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec900245b0 con 0x7feca0102480 2024-01-23T08:43:44.525 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.671+0000 7feca6509700 1 -- 172.21.15.59:0/1331627247 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca0111880 con 0x7feca0102480 2024-01-23T08:43:44.525 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.671+0000 7feca6509700 1 -- 172.21.15.59:0/1331627247 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca0111c90 con 0x7feca0102480 2024-01-23T08:43:44.526 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.672+0000 7fec9cff9700 1 -- 172.21.15.59:0/1331627247 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fec900247a0 con 0x7feca0102480 2024-01-23T08:43:44.526 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.672+0000 7feca6509700 1 -- 172.21.15.59:0/1331627247 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca005f450 con 0x7feca0102480 2024-01-23T08:43:44.526 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.672+0000 7fec9cff9700 1 --2- 172.21.15.59:0/1331627247 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fec88036640 0x7fec88038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:44.526 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.672+0000 7fec9cff9700 1 -- 172.21.15.59:0/1331627247 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fec9004aca0 con 0x7feca0102480 2024-01-23T08:43:44.526 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.673+0000 7fec9f7fe700 1 --2- 172.21.15.59:0/1331627247 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fec88036640 0x7fec88038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:44.526 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.673+0000 7fec9f7fe700 1 --2- 172.21.15.59:0/1331627247 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fec88036640 0x7fec88038af0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fec94009990 tx=0x7fec94006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:44.526 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.677+0000 7fec9cff9700 1 -- 172.21.15.59:0/1331627247 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fec90023800 con 0x7feca0102480 2024-01-23T08:43:44.526 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.888+0000 7feca6509700 1 -- 172.21.15.59:0/1331627247 --> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7feca00ffea0 con 0x7fec88036640 2024-01-23T08:43:44.526 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.896+0000 7fec9cff9700 1 -- 172.21.15.59:0/1331627247 <== mgr.14120 v2:172.21.15.59:6800/1337467262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7feca00ffea0 con 0x7fec88036640 2024-01-23T08:43:44.526 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.898+0000 7feca6509700 1 -- 172.21.15.59:0/1331627247 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fec88036640 msgr2=0x7fec88038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:44.526 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.898+0000 7feca6509700 1 --2- 172.21.15.59:0/1331627247 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fec88036640 0x7fec88038af0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fec94009990 tx=0x7fec94006e40 comp rx=0 tx=0).stop 2024-01-23T08:43:44.526 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.898+0000 7feca6509700 1 -- 172.21.15.59:0/1331627247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca0102480 msgr2=0x7feca0111140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:44.526 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.898+0000 7feca6509700 1 --2- 172.21.15.59:0/1331627247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca0102480 0x7feca0111140 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fec90000c00 tx=0x7fec90009250 comp rx=0 tx=0).stop 2024-01-23T08:43:44.526 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.898+0000 7feca6509700 1 -- 172.21.15.59:0/1331627247 shutdown_connections 2024-01-23T08:43:44.526 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.898+0000 7feca6509700 1 --2- 172.21.15.59:0/1331627247 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fec88036640 0x7fec88038af0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:44.527 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.898+0000 7feca6509700 1 --2- 172.21.15.59:0/1331627247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca0102480 0x7feca0111140 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:44.527 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.898+0000 7feca6509700 1 -- 172.21.15.59:0/1331627247 >> 172.21.15.59:0/1331627247 conn(0x7feca00fdbe0 msgr2=0x7feca00ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:44.527 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.898+0000 7feca6509700 1 -- 172.21.15.59:0/1331627247 shutdown_connections 2024-01-23T08:43:44.527 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:43.898+0000 7feca6509700 1 -- 172.21.15.59:0/1331627247 wait complete. 2024-01-23T08:43:44.527 INFO:teuthology.orchestra.run.smithi059.stdout:Deploying crash service with default placement... 2024-01-23T08:43:45.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:44 smithi059 ceph-mon[28424]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:43:45.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:44 smithi059 ceph-mon[28424]: Saving service mgr spec with placement count:2 2024-01-23T08:43:45.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:44 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' 2024-01-23T08:43:46.551 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:46 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' 2024-01-23T08:43:46.552 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:46 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' 2024-01-23T08:43:46.552 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:46 smithi059 ceph-mon[28424]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:43:46.552 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:46 smithi059 ceph-mon[28424]: Saving service crash spec with placement * 2024-01-23T08:43:46.552 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:46 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' 2024-01-23T08:43:46.565 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-01-23T08:43:46.565 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.764+0000 7fa4006bc700 1 Processor -- start 2024-01-23T08:43:46.565 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.764+0000 7fa4006bc700 1 -- start start 2024-01-23T08:43:46.565 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.764+0000 7fa4006bc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3f8101400 0x7fa3f8101810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:46.566 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.764+0000 7fa4006bc700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f8101de0 con 0x7fa3f8101400 2024-01-23T08:43:46.566 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.764+0000 7fa3fe458700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3f8101400 0x7fa3f8101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:46.566 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.764+0000 7fa3fe458700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3f8101400 0x7fa3f8101810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37146/0 (socket says 172.21.15.59:37146) 2024-01-23T08:43:46.566 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.765+0000 7fa3fe458700 1 -- 172.21.15.59:0/3548509137 learned_addr learned my addr 172.21.15.59:0/3548509137 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:46.566 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.765+0000 7fa3fe458700 1 -- 172.21.15.59:0/3548509137 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f8102610 con 0x7fa3f8101400 2024-01-23T08:43:46.566 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.765+0000 7fa3fe458700 1 --2- 172.21.15.59:0/3548509137 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3f8101400 0x7fa3f8101810 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400bb80 tx=0x7fa3f400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4449ac75708d471c server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:46.566 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.765+0000 7fa3fd456700 1 -- 172.21.15.59:0/3548509137 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3f400dd00 con 0x7fa3f8101400 2024-01-23T08:43:46.567 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.765+0000 7fa3fd456700 1 -- 172.21.15.59:0/3548509137 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3f400de60 con 0x7fa3f8101400 2024-01-23T08:43:46.567 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.765+0000 7fa3fd456700 1 -- 172.21.15.59:0/3548509137 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3f40188d0 con 0x7fa3f8101400 2024-01-23T08:43:46.567 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.766+0000 7fa4006bc700 1 -- 172.21.15.59:0/3548509137 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3f8101400 msgr2=0x7fa3f8101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:46.567 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.766+0000 7fa4006bc700 1 --2- 172.21.15.59:0/3548509137 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3f8101400 0x7fa3f8101810 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400bb80 tx=0x7fa3f400b4f0 comp rx=0 tx=0).stop 2024-01-23T08:43:46.567 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.766+0000 7fa4006bc700 1 -- 172.21.15.59:0/3548509137 shutdown_connections 2024-01-23T08:43:46.567 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.766+0000 7fa4006bc700 1 --2- 172.21.15.59:0/3548509137 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3f8101400 0x7fa3f8101810 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:46.567 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.766+0000 7fa4006bc700 1 -- 172.21.15.59:0/3548509137 >> 172.21.15.59:0/3548509137 conn(0x7fa3f80fcae0 msgr2=0x7fa3f80feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:46.567 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.766+0000 7fa4006bc700 1 -- 172.21.15.59:0/3548509137 shutdown_connections 2024-01-23T08:43:46.568 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.766+0000 7fa4006bc700 1 -- 172.21.15.59:0/3548509137 wait complete. 2024-01-23T08:43:46.568 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.767+0000 7fa4006bc700 1 Processor -- start 2024-01-23T08:43:46.568 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.767+0000 7fa4006bc700 1 -- start start 2024-01-23T08:43:46.568 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.767+0000 7fa4006bc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3f8101400 0x7fa3f810efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:46.568 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.767+0000 7fa4006bc700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f810f520 con 0x7fa3f8101400 2024-01-23T08:43:46.568 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.767+0000 7fa3fe458700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3f8101400 0x7fa3f810efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:46.568 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.767+0000 7fa3fe458700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3f8101400 0x7fa3f810efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37156/0 (socket says 172.21.15.59:37156) 2024-01-23T08:43:46.568 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.767+0000 7fa3fe458700 1 -- 172.21.15.59:0/3715279273 learned_addr learned my addr 172.21.15.59:0/3715279273 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:46.568 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.767+0000 7fa3fe458700 1 -- 172.21.15.59:0/3715279273 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f400b790 con 0x7fa3f8101400 2024-01-23T08:43:46.569 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.767+0000 7fa3fe458700 1 --2- 172.21.15.59:0/3715279273 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3f8101400 0x7fa3f810efe0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400de20 tx=0x7fa3f400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:46.569 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.768+0000 7fa3eeffd700 1 -- 172.21.15.59:0/3715279273 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3f40109d0 con 0x7fa3f8101400 2024-01-23T08:43:46.569 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.768+0000 7fa3eeffd700 1 -- 172.21.15.59:0/3715279273 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3f40223f0 con 0x7fa3f8101400 2024-01-23T08:43:46.569 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.768+0000 7fa4006bc700 1 -- 172.21.15.59:0/3715279273 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f810f720 con 0x7fa3f8101400 2024-01-23T08:43:46.569 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.768+0000 7fa4006bc700 1 -- 172.21.15.59:0/3715279273 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f810fb30 con 0x7fa3f8101400 2024-01-23T08:43:46.569 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.768+0000 7fa3eeffd700 1 -- 172.21.15.59:0/3715279273 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3f4021550 con 0x7fa3f8101400 2024-01-23T08:43:46.569 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.768+0000 7fa3eeffd700 1 -- 172.21.15.59:0/3715279273 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fa3f4021700 con 0x7fa3f8101400 2024-01-23T08:43:46.569 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.769+0000 7fa4006bc700 1 -- 172.21.15.59:0/3715279273 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f8108f50 con 0x7fa3f8101400 2024-01-23T08:43:46.569 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.769+0000 7fa3eeffd700 1 --2- 172.21.15.59:0/3715279273 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fa3e403aae0 0x7fa3e403cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:46.570 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.769+0000 7fa3eeffd700 1 -- 172.21.15.59:0/3715279273 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa3f4014070 con 0x7fa3f8101400 2024-01-23T08:43:46.570 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.769+0000 7fa3fdc57700 1 --2- 172.21.15.59:0/3715279273 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fa3e403aae0 0x7fa3e403cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:46.570 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.769+0000 7fa3fdc57700 1 --2- 172.21.15.59:0/3715279273 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fa3e403aae0 0x7fa3e403cf90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8009990 tx=0x7fa3e8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:46.570 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.774+0000 7fa3eeffd700 1 -- 172.21.15.59:0/3715279273 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa3f400edc0 con 0x7fa3f8101400 2024-01-23T08:43:46.570 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.994+0000 7fa4006bc700 1 -- 172.21.15.59:0/3715279273 --> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3f802b470 con 0x7fa3e403aae0 2024-01-23T08:43:46.570 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:45.999+0000 7fa3eeffd700 1 -- 172.21.15.59:0/3715279273 <== mgr.14120 v2:172.21.15.59:6800/1337467262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fa3f802b470 con 0x7fa3e403aae0 2024-01-23T08:43:46.570 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.001+0000 7fa4006bc700 1 -- 172.21.15.59:0/3715279273 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fa3e403aae0 msgr2=0x7fa3e403cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:46.571 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.001+0000 7fa4006bc700 1 --2- 172.21.15.59:0/3715279273 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fa3e403aae0 0x7fa3e403cf90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8009990 tx=0x7fa3e8006e40 comp rx=0 tx=0).stop 2024-01-23T08:43:46.571 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.001+0000 7fa4006bc700 1 -- 172.21.15.59:0/3715279273 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3f8101400 msgr2=0x7fa3f810efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:46.571 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.001+0000 7fa4006bc700 1 --2- 172.21.15.59:0/3715279273 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3f8101400 0x7fa3f810efe0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400de20 tx=0x7fa3f400b4f0 comp rx=0 tx=0).stop 2024-01-23T08:43:46.571 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.001+0000 7fa4006bc700 1 -- 172.21.15.59:0/3715279273 shutdown_connections 2024-01-23T08:43:46.571 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.001+0000 7fa4006bc700 1 --2- 172.21.15.59:0/3715279273 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fa3e403aae0 0x7fa3e403cf90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:46.571 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.001+0000 7fa4006bc700 1 --2- 172.21.15.59:0/3715279273 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3f8101400 0x7fa3f810efe0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:46.571 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.001+0000 7fa4006bc700 1 -- 172.21.15.59:0/3715279273 >> 172.21.15.59:0/3715279273 conn(0x7fa3f80fcae0 msgr2=0x7fa3f81060d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:46.571 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.001+0000 7fa4006bc700 1 -- 172.21.15.59:0/3715279273 shutdown_connections 2024-01-23T08:43:46.571 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.001+0000 7fa4006bc700 1 -- 172.21.15.59:0/3715279273 wait complete. 2024-01-23T08:43:46.571 INFO:teuthology.orchestra.run.smithi059.stdout:Deploying prometheus service with default placement... 2024-01-23T08:43:48.227 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-01-23T08:43:48.228 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.952+0000 7f0fe395a700 1 Processor -- start 2024-01-23T08:43:48.228 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.952+0000 7f0fe395a700 1 -- start start 2024-01-23T08:43:48.228 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.952+0000 7f0fe395a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fdc076ca0 0x7f0fdc1074b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:48.228 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.952+0000 7f0fe395a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fdc102f30 con 0x7f0fdc076ca0 2024-01-23T08:43:48.228 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.952+0000 7f0fe16f6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fdc076ca0 0x7f0fdc1074b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:48.228 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.953+0000 7f0fe16f6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fdc076ca0 0x7f0fdc1074b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37168/0 (socket says 172.21.15.59:37168) 2024-01-23T08:43:48.228 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.953+0000 7f0fe16f6700 1 -- 172.21.15.59:0/1141574042 learned_addr learned my addr 172.21.15.59:0/1141574042 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:48.228 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.953+0000 7f0fe16f6700 1 -- 172.21.15.59:0/1141574042 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fdc1079f0 con 0x7f0fdc076ca0 2024-01-23T08:43:48.228 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.953+0000 7f0fe16f6700 1 --2- 172.21.15.59:0/1141574042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fdc076ca0 0x7f0fdc1074b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0fd40098e0 tx=0x7f0fd4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac449704e51df457 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:48.228 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.953+0000 7f0fdbfff700 1 -- 172.21.15.59:0/1141574042 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fd400b860 con 0x7f0fdc076ca0 2024-01-23T08:43:48.228 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.953+0000 7f0fdbfff700 1 -- 172.21.15.59:0/1141574042 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0fd400be70 con 0x7f0fdc076ca0 2024-01-23T08:43:48.228 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.953+0000 7f0fdbfff700 1 -- 172.21.15.59:0/1141574042 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fd4010380 con 0x7f0fdc076ca0 2024-01-23T08:43:48.229 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.954+0000 7f0fe395a700 1 -- 172.21.15.59:0/1141574042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fdc076ca0 msgr2=0x7f0fdc1074b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:48.229 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.954+0000 7f0fe395a700 1 --2- 172.21.15.59:0/1141574042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fdc076ca0 0x7f0fdc1074b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0fd40098e0 tx=0x7f0fd4009250 comp rx=0 tx=0).stop 2024-01-23T08:43:48.229 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.954+0000 7f0fe395a700 1 -- 172.21.15.59:0/1141574042 shutdown_connections 2024-01-23T08:43:48.229 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.954+0000 7f0fe395a700 1 --2- 172.21.15.59:0/1141574042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fdc076ca0 0x7f0fdc1074b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:48.229 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.954+0000 7f0fe395a700 1 -- 172.21.15.59:0/1141574042 >> 172.21.15.59:0/1141574042 conn(0x7f0fdc0fcae0 msgr2=0x7f0fdc0feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:48.229 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.954+0000 7f0fe395a700 1 -- 172.21.15.59:0/1141574042 shutdown_connections 2024-01-23T08:43:48.229 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.954+0000 7f0fe395a700 1 -- 172.21.15.59:0/1141574042 wait complete. 2024-01-23T08:43:48.229 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.954+0000 7f0fe395a700 1 Processor -- start 2024-01-23T08:43:48.229 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.954+0000 7f0fe395a700 1 -- start start 2024-01-23T08:43:48.229 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.954+0000 7f0fe395a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fdc076ca0 0x7f0fdc106970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:48.229 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.955+0000 7f0fe395a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fdc106eb0 con 0x7f0fdc076ca0 2024-01-23T08:43:48.229 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.955+0000 7f0fe16f6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fdc076ca0 0x7f0fdc106970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:48.229 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.955+0000 7f0fe16f6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fdc076ca0 0x7f0fdc106970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37180/0 (socket says 172.21.15.59:37180) 2024-01-23T08:43:48.229 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.955+0000 7f0fe16f6700 1 -- 172.21.15.59:0/652160182 learned_addr learned my addr 172.21.15.59:0/652160182 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:48.230 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.955+0000 7f0fe16f6700 1 -- 172.21.15.59:0/652160182 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd40094f0 con 0x7f0fdc076ca0 2024-01-23T08:43:48.230 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.955+0000 7f0fe16f6700 1 --2- 172.21.15.59:0/652160182 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fdc076ca0 0x7f0fdc106970 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0fd400b440 tx=0x7f0fd4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:48.230 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.955+0000 7f0fd9ffb700 1 -- 172.21.15.59:0/652160182 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fd4010850 con 0x7f0fdc076ca0 2024-01-23T08:43:48.230 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.955+0000 7f0fd9ffb700 1 -- 172.21.15.59:0/652160182 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0fd4010e60 con 0x7f0fdc076ca0 2024-01-23T08:43:48.230 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.955+0000 7f0fd9ffb700 1 -- 172.21.15.59:0/652160182 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fd401ae60 con 0x7f0fdc076ca0 2024-01-23T08:43:48.230 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.955+0000 7f0fe395a700 1 -- 172.21.15.59:0/652160182 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fdc1070b0 con 0x7f0fdc076ca0 2024-01-23T08:43:48.230 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.956+0000 7f0fe395a700 1 -- 172.21.15.59:0/652160182 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fdc103580 con 0x7f0fdc076ca0 2024-01-23T08:43:48.230 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.956+0000 7f0fd9ffb700 1 -- 172.21.15.59:0/652160182 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f0fd401a430 con 0x7f0fdc076ca0 2024-01-23T08:43:48.230 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.956+0000 7f0fd9ffb700 1 --2- 172.21.15.59:0/652160182 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f0fc4036640 0x7f0fc4038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:48.230 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.957+0000 7f0fd9ffb700 1 -- 172.21.15.59:0/652160182 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f0fd404a560 con 0x7f0fdc076ca0 2024-01-23T08:43:48.230 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.957+0000 7f0fe0ef5700 1 --2- 172.21.15.59:0/652160182 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f0fc4036640 0x7f0fc4038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:48.230 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.957+0000 7f0fe395a700 1 -- 172.21.15.59:0/652160182 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fdc05f480 con 0x7f0fdc076ca0 2024-01-23T08:43:48.230 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.957+0000 7f0fe0ef5700 1 --2- 172.21.15.59:0/652160182 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f0fc4036640 0x7f0fc4038af0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0009990 tx=0x7f0fd0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:48.230 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:46.962+0000 7f0fd9ffb700 1 -- 172.21.15.59:0/652160182 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0fd401a710 con 0x7f0fdc076ca0 2024-01-23T08:43:48.231 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:47.169+0000 7f0fe395a700 1 -- 172.21.15.59:0/652160182 --> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fdc0fedd0 con 0x7f0fc4036640 2024-01-23T08:43:48.231 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:47.173+0000 7f0fd9ffb700 1 -- 172.21.15.59:0/652160182 <== mgr.14120 v2:172.21.15.59:6800/1337467262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f0fdc0fedd0 con 0x7f0fc4036640 2024-01-23T08:43:48.231 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:47.175+0000 7f0fe395a700 1 -- 172.21.15.59:0/652160182 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f0fc4036640 msgr2=0x7f0fc4038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:48.231 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:47.175+0000 7f0fe395a700 1 --2- 172.21.15.59:0/652160182 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f0fc4036640 0x7f0fc4038af0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0009990 tx=0x7f0fd0006e40 comp rx=0 tx=0).stop 2024-01-23T08:43:48.231 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:47.175+0000 7f0fe395a700 1 -- 172.21.15.59:0/652160182 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fdc076ca0 msgr2=0x7f0fdc106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:48.231 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:47.175+0000 7f0fe395a700 1 --2- 172.21.15.59:0/652160182 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fdc076ca0 0x7f0fdc106970 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0fd400b440 tx=0x7f0fd4009250 comp rx=0 tx=0).stop 2024-01-23T08:43:48.231 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:47.175+0000 7f0fe395a700 1 -- 172.21.15.59:0/652160182 shutdown_connections 2024-01-23T08:43:48.231 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:47.175+0000 7f0fe395a700 1 --2- 172.21.15.59:0/652160182 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f0fc4036640 0x7f0fc4038af0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:48.231 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:47.175+0000 7f0fe395a700 1 --2- 172.21.15.59:0/652160182 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fdc076ca0 0x7f0fdc106970 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:48.231 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:47.175+0000 7f0fe395a700 1 -- 172.21.15.59:0/652160182 >> 172.21.15.59:0/652160182 conn(0x7f0fdc0fcae0 msgr2=0x7f0fdc0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:48.231 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:47.176+0000 7f0fe395a700 1 -- 172.21.15.59:0/652160182 shutdown_connections 2024-01-23T08:43:48.231 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:47.176+0000 7f0fe395a700 1 -- 172.21.15.59:0/652160182 wait complete. 2024-01-23T08:43:48.231 INFO:teuthology.orchestra.run.smithi059.stdout:Deploying grafana service with default placement... 2024-01-23T08:43:48.462 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:48 smithi059 ceph-mon[28424]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:43:48.463 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:48 smithi059 ceph-mon[28424]: Saving service prometheus spec with placement count:1 2024-01-23T08:43:48.463 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:48 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' 2024-01-23T08:43:49.510 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-01-23T08:43:49.510 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.650+0000 7f2c48167700 1 Processor -- start 2024-01-23T08:43:49.510 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.651+0000 7f2c48167700 1 -- start start 2024-01-23T08:43:49.511 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.651+0000 7f2c48167700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c40102450 0x7f2c40102860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:49.511 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.651+0000 7f2c48167700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c40102da0 con 0x7f2c40102450 2024-01-23T08:43:49.511 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.651+0000 7f2c45f03700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c40102450 0x7f2c40102860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:49.511 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.651+0000 7f2c45f03700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c40102450 0x7f2c40102860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34460/0 (socket says 172.21.15.59:34460) 2024-01-23T08:43:49.511 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.651+0000 7f2c45f03700 1 -- 172.21.15.59:0/76420070 learned_addr learned my addr 172.21.15.59:0/76420070 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:49.511 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.651+0000 7f2c45f03700 1 -- 172.21.15.59:0/76420070 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c40102ee0 con 0x7f2c40102450 2024-01-23T08:43:49.511 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.651+0000 7f2c45f03700 1 --2- 172.21.15.59:0/76420070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c40102450 0x7f2c40102860 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2c380098e0 tx=0x7f2c38009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e3cde59e361bb47 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:49.511 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.651+0000 7f2c44f01700 1 -- 172.21.15.59:0/76420070 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c3800b860 con 0x7f2c40102450 2024-01-23T08:43:49.511 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.651+0000 7f2c44f01700 1 -- 172.21.15.59:0/76420070 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c3800be70 con 0x7f2c40102450 2024-01-23T08:43:49.511 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.651+0000 7f2c44f01700 1 -- 172.21.15.59:0/76420070 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c38010380 con 0x7f2c40102450 2024-01-23T08:43:49.511 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.652+0000 7f2c48167700 1 -- 172.21.15.59:0/76420070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c40102450 msgr2=0x7f2c40102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:49.511 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.652+0000 7f2c48167700 1 --2- 172.21.15.59:0/76420070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c40102450 0x7f2c40102860 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2c380098e0 tx=0x7f2c38009250 comp rx=0 tx=0).stop 2024-01-23T08:43:49.511 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.652+0000 7f2c48167700 1 -- 172.21.15.59:0/76420070 shutdown_connections 2024-01-23T08:43:49.512 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.652+0000 7f2c48167700 1 --2- 172.21.15.59:0/76420070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c40102450 0x7f2c40102860 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:49.512 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.652+0000 7f2c48167700 1 -- 172.21.15.59:0/76420070 >> 172.21.15.59:0/76420070 conn(0x7f2c400fdbd0 msgr2=0x7f2c400fffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:49.512 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.652+0000 7f2c48167700 1 -- 172.21.15.59:0/76420070 shutdown_connections 2024-01-23T08:43:49.512 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.652+0000 7f2c48167700 1 -- 172.21.15.59:0/76420070 wait complete. 2024-01-23T08:43:49.512 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.653+0000 7f2c48167700 1 Processor -- start 2024-01-23T08:43:49.512 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.653+0000 7f2c48167700 1 -- start start 2024-01-23T08:43:49.512 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.653+0000 7f2c48167700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c40102450 0x7f2c40111160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:49.512 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.653+0000 7f2c48167700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c401116a0 con 0x7f2c40102450 2024-01-23T08:43:49.512 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.653+0000 7f2c45f03700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c40102450 0x7f2c40111160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:49.512 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.653+0000 7f2c45f03700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c40102450 0x7f2c40111160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34462/0 (socket says 172.21.15.59:34462) 2024-01-23T08:43:49.512 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.653+0000 7f2c45f03700 1 -- 172.21.15.59:0/816346940 learned_addr learned my addr 172.21.15.59:0/816346940 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:49.512 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.653+0000 7f2c45f03700 1 -- 172.21.15.59:0/816346940 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c380094f0 con 0x7f2c40102450 2024-01-23T08:43:49.512 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.653+0000 7f2c45f03700 1 --2- 172.21.15.59:0/816346940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c40102450 0x7f2c40111160 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2c3800bac0 tx=0x7f2c38009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:49.513 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.653+0000 7f2c3e7fc700 1 -- 172.21.15.59:0/816346940 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c380109b0 con 0x7f2c40102450 2024-01-23T08:43:49.513 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.653+0000 7f2c3e7fc700 1 -- 172.21.15.59:0/816346940 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c3801a430 con 0x7f2c40102450 2024-01-23T08:43:49.513 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.653+0000 7f2c48167700 1 -- 172.21.15.59:0/816346940 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c401118a0 con 0x7f2c40102450 2024-01-23T08:43:49.513 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.654+0000 7f2c3e7fc700 1 -- 172.21.15.59:0/816346940 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c380245b0 con 0x7f2c40102450 2024-01-23T08:43:49.513 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.654+0000 7f2c48167700 1 -- 172.21.15.59:0/816346940 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c40111cb0 con 0x7f2c40102450 2024-01-23T08:43:49.513 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.654+0000 7f2c3e7fc700 1 -- 172.21.15.59:0/816346940 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f2c3801a5a0 con 0x7f2c40102450 2024-01-23T08:43:49.513 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.654+0000 7f2c3e7fc700 1 --2- 172.21.15.59:0/816346940 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f2c28036640 0x7f2c28038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:49.513 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.654+0000 7f2c48167700 1 -- 172.21.15.59:0/816346940 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c4005f450 con 0x7f2c40102450 2024-01-23T08:43:49.513 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.655+0000 7f2c3e7fc700 1 -- 172.21.15.59:0/816346940 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f2c3804a520 con 0x7f2c40102450 2024-01-23T08:43:49.513 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.655+0000 7f2c45702700 1 --2- 172.21.15.59:0/816346940 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f2c28036640 0x7f2c28038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:49.513 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.655+0000 7f2c45702700 1 --2- 172.21.15.59:0/816346940 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f2c28036640 0x7f2c28038af0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2c34009990 tx=0x7f2c34006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:49.513 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.660+0000 7f2c3e7fc700 1 -- 172.21.15.59:0/816346940 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2c38010b10 con 0x7f2c40102450 2024-01-23T08:43:49.513 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.872+0000 7f2c48167700 1 -- 172.21.15.59:0/816346940 --> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c400ffe80 con 0x7f2c28036640 2024-01-23T08:43:49.514 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.878+0000 7f2c3e7fc700 1 -- 172.21.15.59:0/816346940 <== mgr.14120 v2:172.21.15.59:6800/1337467262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f2c400ffe80 con 0x7f2c28036640 2024-01-23T08:43:49.514 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.880+0000 7f2c48167700 1 -- 172.21.15.59:0/816346940 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f2c28036640 msgr2=0x7f2c28038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:49.514 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.880+0000 7f2c48167700 1 --2- 172.21.15.59:0/816346940 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f2c28036640 0x7f2c28038af0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2c34009990 tx=0x7f2c34006e40 comp rx=0 tx=0).stop 2024-01-23T08:43:49.514 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.880+0000 7f2c48167700 1 -- 172.21.15.59:0/816346940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c40102450 msgr2=0x7f2c40111160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:49.514 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.880+0000 7f2c48167700 1 --2- 172.21.15.59:0/816346940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c40102450 0x7f2c40111160 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2c3800bac0 tx=0x7f2c38009250 comp rx=0 tx=0).stop 2024-01-23T08:43:49.514 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.880+0000 7f2c48167700 1 -- 172.21.15.59:0/816346940 shutdown_connections 2024-01-23T08:43:49.514 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.880+0000 7f2c48167700 1 --2- 172.21.15.59:0/816346940 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f2c28036640 0x7f2c28038af0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:49.514 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.880+0000 7f2c48167700 1 --2- 172.21.15.59:0/816346940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c40102450 0x7f2c40111160 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:49.514 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.880+0000 7f2c48167700 1 -- 172.21.15.59:0/816346940 >> 172.21.15.59:0/816346940 conn(0x7f2c400fdbd0 msgr2=0x7f2c400ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:49.514 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.880+0000 7f2c48167700 1 -- 172.21.15.59:0/816346940 shutdown_connections 2024-01-23T08:43:49.514 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:48.880+0000 7f2c48167700 1 -- 172.21.15.59:0/816346940 wait complete. 2024-01-23T08:43:49.514 INFO:teuthology.orchestra.run.smithi059.stdout:Deploying node-exporter service with default placement... 2024-01-23T08:43:50.195 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:49 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' 2024-01-23T08:43:50.195 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:49 smithi059 ceph-mon[28424]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:43:50.195 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:49 smithi059 ceph-mon[28424]: Saving service grafana spec with placement count:1 2024-01-23T08:43:50.195 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:49 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' 2024-01-23T08:43:50.195 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:49 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' 2024-01-23T08:43:50.856 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-01-23T08:43:50.856 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.980+0000 7f82b9a57700 1 Processor -- start 2024-01-23T08:43:50.856 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.980+0000 7f82b9a57700 1 -- start start 2024-01-23T08:43:50.856 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.981+0000 7f82b9a57700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b4102500 0x7f82b4102910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:50.856 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.981+0000 7f82b9a57700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b4102e50 con 0x7f82b4102500 2024-01-23T08:43:50.856 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.981+0000 7f82b2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b4102500 0x7f82b4102910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:50.856 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.981+0000 7f82b2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b4102500 0x7f82b4102910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34476/0 (socket says 172.21.15.59:34476) 2024-01-23T08:43:50.856 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.981+0000 7f82b2ffd700 1 -- 172.21.15.59:0/1084028339 learned_addr learned my addr 172.21.15.59:0/1084028339 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:50.856 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.981+0000 7f82b2ffd700 1 -- 172.21.15.59:0/1084028339 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b4102f90 con 0x7f82b4102500 2024-01-23T08:43:50.856 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.981+0000 7f82b2ffd700 1 --2- 172.21.15.59:0/1084028339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b4102500 0x7f82b4102910 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f82a40098e0 tx=0x7f82a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75bf17685238febb server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:50.856 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.981+0000 7f82b1ffb700 1 -- 172.21.15.59:0/1084028339 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82a400b860 con 0x7f82b4102500 2024-01-23T08:43:50.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.981+0000 7f82b1ffb700 1 -- 172.21.15.59:0/1084028339 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f82a400be70 con 0x7f82b4102500 2024-01-23T08:43:50.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.981+0000 7f82b1ffb700 1 -- 172.21.15.59:0/1084028339 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82a4010380 con 0x7f82b4102500 2024-01-23T08:43:50.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.982+0000 7f82b9a57700 1 -- 172.21.15.59:0/1084028339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b4102500 msgr2=0x7f82b4102910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:50.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.982+0000 7f82b9a57700 1 --2- 172.21.15.59:0/1084028339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b4102500 0x7f82b4102910 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f82a40098e0 tx=0x7f82a4009250 comp rx=0 tx=0).stop 2024-01-23T08:43:50.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.982+0000 7f82b9a57700 1 -- 172.21.15.59:0/1084028339 shutdown_connections 2024-01-23T08:43:50.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.982+0000 7f82b9a57700 1 --2- 172.21.15.59:0/1084028339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b4102500 0x7f82b4102910 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:50.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.982+0000 7f82b9a57700 1 -- 172.21.15.59:0/1084028339 >> 172.21.15.59:0/1084028339 conn(0x7f82b40fdca0 msgr2=0x7f82b41000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:50.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.982+0000 7f82b9a57700 1 -- 172.21.15.59:0/1084028339 shutdown_connections 2024-01-23T08:43:50.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.982+0000 7f82b9a57700 1 -- 172.21.15.59:0/1084028339 wait complete. 2024-01-23T08:43:50.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.983+0000 7f82b9a57700 1 Processor -- start 2024-01-23T08:43:50.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.983+0000 7f82b9a57700 1 -- start start 2024-01-23T08:43:50.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.983+0000 7f82b9a57700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b4102500 0x7f82b41114d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:50.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.983+0000 7f82b9a57700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b4111a10 con 0x7f82b4102500 2024-01-23T08:43:50.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.983+0000 7f82b2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b4102500 0x7f82b41114d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:50.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.983+0000 7f82b2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b4102500 0x7f82b41114d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34492/0 (socket says 172.21.15.59:34492) 2024-01-23T08:43:50.858 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.983+0000 7f82b2ffd700 1 -- 172.21.15.59:0/1771694992 learned_addr learned my addr 172.21.15.59:0/1771694992 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:50.858 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.983+0000 7f82b2ffd700 1 -- 172.21.15.59:0/1771694992 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a40094f0 con 0x7f82b4102500 2024-01-23T08:43:50.858 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.984+0000 7f82b2ffd700 1 --2- 172.21.15.59:0/1771694992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b4102500 0x7f82b41114d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f82a4003080 tx=0x7f82a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:50.858 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.984+0000 7f829b7fe700 1 -- 172.21.15.59:0/1771694992 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82a4010800 con 0x7f82b4102500 2024-01-23T08:43:50.858 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.984+0000 7f829b7fe700 1 -- 172.21.15.59:0/1771694992 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f82a4010e10 con 0x7f82b4102500 2024-01-23T08:43:50.858 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.984+0000 7f82b9a57700 1 -- 172.21.15.59:0/1771694992 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b4111c10 con 0x7f82b4102500 2024-01-23T08:43:50.858 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.984+0000 7f829b7fe700 1 -- 172.21.15.59:0/1771694992 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82a401a430 con 0x7f82b4102500 2024-01-23T08:43:50.858 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.984+0000 7f82b9a57700 1 -- 172.21.15.59:0/1771694992 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b4112020 con 0x7f82b4102500 2024-01-23T08:43:50.858 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.985+0000 7f829b7fe700 1 -- 172.21.15.59:0/1771694992 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f82a401a660 con 0x7f82b4102500 2024-01-23T08:43:50.858 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.985+0000 7f82b9a57700 1 -- 172.21.15.59:0/1771694992 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b410b3c0 con 0x7f82b4102500 2024-01-23T08:43:50.858 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.985+0000 7f829b7fe700 1 --2- 172.21.15.59:0/1771694992 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f829c036640 0x7f829c038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:50.858 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.985+0000 7f829b7fe700 1 -- 172.21.15.59:0/1771694992 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f82a404a6a0 con 0x7f82b4102500 2024-01-23T08:43:50.858 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.985+0000 7f82b27fc700 1 --2- 172.21.15.59:0/1771694992 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f829c036640 0x7f829c038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:50.858 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.985+0000 7f82b27fc700 1 --2- 172.21.15.59:0/1771694992 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f829c036640 0x7f829c038af0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f82a8009990 tx=0x7f82a8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:50.858 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:49.990+0000 7f829b7fe700 1 -- 172.21.15.59:0/1771694992 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f82a4018b40 con 0x7f82b4102500 2024-01-23T08:43:50.859 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:50.197+0000 7f82b9a57700 1 -- 172.21.15.59:0/1771694992 --> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f82b405e260 con 0x7f829c036640 2024-01-23T08:43:50.859 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:50.201+0000 7f829b7fe700 1 -- 172.21.15.59:0/1771694992 <== mgr.14120 v2:172.21.15.59:6800/1337467262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f82b405e260 con 0x7f829c036640 2024-01-23T08:43:50.859 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:50.203+0000 7f82b9a57700 1 -- 172.21.15.59:0/1771694992 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f829c036640 msgr2=0x7f829c038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:50.859 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:50.203+0000 7f82b9a57700 1 --2- 172.21.15.59:0/1771694992 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f829c036640 0x7f829c038af0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f82a8009990 tx=0x7f82a8006e40 comp rx=0 tx=0).stop 2024-01-23T08:43:50.859 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:50.203+0000 7f82b9a57700 1 -- 172.21.15.59:0/1771694992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b4102500 msgr2=0x7f82b41114d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:50.859 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:50.203+0000 7f82b9a57700 1 --2- 172.21.15.59:0/1771694992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b4102500 0x7f82b41114d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f82a4003080 tx=0x7f82a4009250 comp rx=0 tx=0).stop 2024-01-23T08:43:50.859 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:50.203+0000 7f82b9a57700 1 -- 172.21.15.59:0/1771694992 shutdown_connections 2024-01-23T08:43:50.859 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:50.203+0000 7f82b9a57700 1 --2- 172.21.15.59:0/1771694992 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f829c036640 0x7f829c038af0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:50.859 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:50.203+0000 7f82b9a57700 1 --2- 172.21.15.59:0/1771694992 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b4102500 0x7f82b41114d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:50.859 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:50.204+0000 7f82b9a57700 1 -- 172.21.15.59:0/1771694992 >> 172.21.15.59:0/1771694992 conn(0x7f82b40fdca0 msgr2=0x7f82b40feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:50.859 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:50.204+0000 7f82b9a57700 1 -- 172.21.15.59:0/1771694992 shutdown_connections 2024-01-23T08:43:50.859 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:50.204+0000 7f82b9a57700 1 -- 172.21.15.59:0/1771694992 wait complete. 2024-01-23T08:43:50.859 INFO:teuthology.orchestra.run.smithi059.stdout:Deploying alertmanager service with default placement... 2024-01-23T08:43:51.405 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:51 smithi059 ceph-mon[28424]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:43:51.405 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:51 smithi059 ceph-mon[28424]: Saving service node-exporter spec with placement * 2024-01-23T08:43:51.405 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:51 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' 2024-01-23T08:43:52.963 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:52 smithi059 ceph-mon[28424]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:43:52.963 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:52 smithi059 ceph-mon[28424]: from='mgr.14120 172.21.15.59:0/2378806099' entity='mgr.smithi059.bzqirf' 2024-01-23T08:43:52.970 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-01-23T08:43:52.970 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.137+0000 7fc15ee46700 1 Processor -- start 2024-01-23T08:43:52.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.137+0000 7fc15ee46700 1 -- start start 2024-01-23T08:43:52.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.137+0000 7fc15ee46700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1581013f0 0x7fc158101800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:52.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.137+0000 7fc15ee46700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc158101dd0 con 0x7fc1581013f0 2024-01-23T08:43:52.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.138+0000 7fc15cbe2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1581013f0 0x7fc158101800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:52.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.138+0000 7fc15cbe2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1581013f0 0x7fc158101800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34508/0 (socket says 172.21.15.59:34508) 2024-01-23T08:43:52.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.138+0000 7fc15cbe2700 1 -- 172.21.15.59:0/1079528204 learned_addr learned my addr 172.21.15.59:0/1079528204 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:52.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.138+0000 7fc15cbe2700 1 -- 172.21.15.59:0/1079528204 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc158102600 con 0x7fc1581013f0 2024-01-23T08:43:52.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.138+0000 7fc15cbe2700 1 --2- 172.21.15.59:0/1079528204 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1581013f0 0x7fc158101800 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc150009e50 tx=0x7fc150009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c75a173839efbb29 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:52.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.138+0000 7fc1577fe700 1 -- 172.21.15.59:0/1079528204 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1500092e0 con 0x7fc1581013f0 2024-01-23T08:43:52.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.138+0000 7fc1577fe700 1 -- 172.21.15.59:0/1079528204 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc150012430 con 0x7fc1581013f0 2024-01-23T08:43:52.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.138+0000 7fc1577fe700 1 -- 172.21.15.59:0/1079528204 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1500075c0 con 0x7fc1581013f0 2024-01-23T08:43:52.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.139+0000 7fc15ee46700 1 -- 172.21.15.59:0/1079528204 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1581013f0 msgr2=0x7fc158101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:52.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.139+0000 7fc15ee46700 1 --2- 172.21.15.59:0/1079528204 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1581013f0 0x7fc158101800 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc150009e50 tx=0x7fc150009840 comp rx=0 tx=0).stop 2024-01-23T08:43:52.971 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.139+0000 7fc15ee46700 1 -- 172.21.15.59:0/1079528204 shutdown_connections 2024-01-23T08:43:52.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.139+0000 7fc15ee46700 1 --2- 172.21.15.59:0/1079528204 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1581013f0 0x7fc158101800 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:52.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.139+0000 7fc15ee46700 1 -- 172.21.15.59:0/1079528204 >> 172.21.15.59:0/1079528204 conn(0x7fc1580fcab0 msgr2=0x7fc1580feee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:52.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.139+0000 7fc15ee46700 1 -- 172.21.15.59:0/1079528204 shutdown_connections 2024-01-23T08:43:52.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.139+0000 7fc15ee46700 1 -- 172.21.15.59:0/1079528204 wait complete. 2024-01-23T08:43:52.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.140+0000 7fc15ee46700 1 Processor -- start 2024-01-23T08:43:52.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.140+0000 7fc15ee46700 1 -- start start 2024-01-23T08:43:52.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.140+0000 7fc15ee46700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1581013f0 0x7fc158070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:52.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.140+0000 7fc15ee46700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc158114010 con 0x7fc1581013f0 2024-01-23T08:43:52.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.140+0000 7fc15cbe2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1581013f0 0x7fc158070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:52.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.140+0000 7fc15cbe2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1581013f0 0x7fc158070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34512/0 (socket says 172.21.15.59:34512) 2024-01-23T08:43:52.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.140+0000 7fc15cbe2700 1 -- 172.21.15.59:0/194104646 learned_addr learned my addr 172.21.15.59:0/194104646 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:52.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.140+0000 7fc15cbe2700 1 -- 172.21.15.59:0/194104646 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc150009ae0 con 0x7fc1581013f0 2024-01-23T08:43:52.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.141+0000 7fc15cbe2700 1 --2- 172.21.15.59:0/194104646 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1581013f0 0x7fc158070ff0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc150009fd0 tx=0x7fc150009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:52.972 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.141+0000 7fc1557fa700 1 -- 172.21.15.59:0/194104646 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1500092e0 con 0x7fc1581013f0 2024-01-23T08:43:52.973 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.141+0000 7fc1557fa700 1 -- 172.21.15.59:0/194104646 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc150007ce0 con 0x7fc1581013f0 2024-01-23T08:43:52.973 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.141+0000 7fc1557fa700 1 -- 172.21.15.59:0/194104646 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc15001c7f0 con 0x7fc1581013f0 2024-01-23T08:43:52.973 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.141+0000 7fc15ee46700 1 -- 172.21.15.59:0/194104646 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc158114150 con 0x7fc1581013f0 2024-01-23T08:43:52.973 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.141+0000 7fc15ee46700 1 -- 172.21.15.59:0/194104646 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc158114500 con 0x7fc1581013f0 2024-01-23T08:43:52.973 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.142+0000 7fc1557fa700 1 -- 172.21.15.59:0/194104646 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fc150019070 con 0x7fc1581013f0 2024-01-23T08:43:52.973 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.142+0000 7fc1557fa700 1 --2- 172.21.15.59:0/194104646 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc144036640 0x7fc144038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:52.973 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.142+0000 7fc1557fa700 1 -- 172.21.15.59:0/194104646 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fc150049cc0 con 0x7fc1581013f0 2024-01-23T08:43:52.973 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.142+0000 7fc15ee46700 1 -- 172.21.15.59:0/194104646 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc158103390 con 0x7fc1581013f0 2024-01-23T08:43:52.973 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.142+0000 7fc157fff700 1 --2- 172.21.15.59:0/194104646 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc144036640 0x7fc144038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:52.973 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.142+0000 7fc157fff700 1 --2- 172.21.15.59:0/194104646 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc144036640 0x7fc144038af0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc148009990 tx=0x7fc148006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:52.973 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.147+0000 7fc1557fa700 1 -- 172.21.15.59:0/194104646 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc150017020 con 0x7fc1581013f0 2024-01-23T08:43:52.973 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.350+0000 7fc15ee46700 1 -- 172.21.15.59:0/194104646 --> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1580fee00 con 0x7fc144036640 2024-01-23T08:43:52.973 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.354+0000 7fc1557fa700 1 -- 172.21.15.59:0/194104646 <== mgr.14120 v2:172.21.15.59:6800/1337467262 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fc1580fee00 con 0x7fc144036640 2024-01-23T08:43:52.973 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.356+0000 7fc15ee46700 1 -- 172.21.15.59:0/194104646 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc144036640 msgr2=0x7fc144038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:52.973 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.356+0000 7fc15ee46700 1 --2- 172.21.15.59:0/194104646 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc144036640 0x7fc144038af0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc148009990 tx=0x7fc148006e40 comp rx=0 tx=0).stop 2024-01-23T08:43:52.974 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.356+0000 7fc15ee46700 1 -- 172.21.15.59:0/194104646 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1581013f0 msgr2=0x7fc158070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:52.974 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.356+0000 7fc15ee46700 1 --2- 172.21.15.59:0/194104646 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1581013f0 0x7fc158070ff0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc150009fd0 tx=0x7fc150009840 comp rx=0 tx=0).stop 2024-01-23T08:43:52.974 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.356+0000 7fc15ee46700 1 -- 172.21.15.59:0/194104646 shutdown_connections 2024-01-23T08:43:52.974 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.356+0000 7fc15ee46700 1 --2- 172.21.15.59:0/194104646 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc144036640 0x7fc144038af0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:52.974 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.356+0000 7fc15ee46700 1 --2- 172.21.15.59:0/194104646 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1581013f0 0x7fc158070ff0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:52.974 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.356+0000 7fc15ee46700 1 -- 172.21.15.59:0/194104646 >> 172.21.15.59:0/194104646 conn(0x7fc1580fcab0 msgr2=0x7fc1580fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:52.974 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.356+0000 7fc15ee46700 1 -- 172.21.15.59:0/194104646 shutdown_connections 2024-01-23T08:43:52.974 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:52.356+0000 7fc15ee46700 1 -- 172.21.15.59:0/194104646 wait complete. 2024-01-23T08:43:53.936 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:53 smithi059 ceph-mon[28424]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:43:53.936 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:53 smithi059 ceph-mon[28424]: Saving service alertmanager spec with placement count:1 2024-01-23T08:43:53.936 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:53 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/2851367415' entity='client.admin' 2024-01-23T08:43:54.245 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.408+0000 7ff314472700 1 Processor -- start 2024-01-23T08:43:54.245 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.409+0000 7ff314472700 1 -- start start 2024-01-23T08:43:54.245 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.409+0000 7ff314472700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff30c102500 0x7ff30c102910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:54.245 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.409+0000 7ff314472700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c102e50 con 0x7ff30c102500 2024-01-23T08:43:54.245 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.409+0000 7ff31220e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff30c102500 0x7ff30c102910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:54.245 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.409+0000 7ff31220e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff30c102500 0x7ff30c102910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34522/0 (socket says 172.21.15.59:34522) 2024-01-23T08:43:54.246 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.409+0000 7ff31220e700 1 -- 172.21.15.59:0/50751428 learned_addr learned my addr 172.21.15.59:0/50751428 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:54.246 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.409+0000 7ff31220e700 1 -- 172.21.15.59:0/50751428 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30c102f90 con 0x7ff30c102500 2024-01-23T08:43:54.246 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.409+0000 7ff31220e700 1 --2- 172.21.15.59:0/50751428 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff30c102500 0x7ff30c102910 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff3080098e0 tx=0x7ff308009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eb879ea47b8aba3 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:54.246 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.410+0000 7ff31120c700 1 -- 172.21.15.59:0/50751428 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff30800b860 con 0x7ff30c102500 2024-01-23T08:43:54.246 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.410+0000 7ff31120c700 1 -- 172.21.15.59:0/50751428 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff30800be70 con 0x7ff30c102500 2024-01-23T08:43:54.246 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.410+0000 7ff31120c700 1 -- 172.21.15.59:0/50751428 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff308010380 con 0x7ff30c102500 2024-01-23T08:43:54.246 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.410+0000 7ff314472700 1 -- 172.21.15.59:0/50751428 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff30c102500 msgr2=0x7ff30c102910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:54.246 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.410+0000 7ff314472700 1 --2- 172.21.15.59:0/50751428 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff30c102500 0x7ff30c102910 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff3080098e0 tx=0x7ff308009250 comp rx=0 tx=0).stop 2024-01-23T08:43:54.246 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.410+0000 7ff314472700 1 -- 172.21.15.59:0/50751428 shutdown_connections 2024-01-23T08:43:54.246 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.410+0000 7ff314472700 1 --2- 172.21.15.59:0/50751428 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff30c102500 0x7ff30c102910 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:54.246 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.410+0000 7ff314472700 1 -- 172.21.15.59:0/50751428 >> 172.21.15.59:0/50751428 conn(0x7ff30c0fdca0 msgr2=0x7ff30c1000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:54.246 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.410+0000 7ff314472700 1 -- 172.21.15.59:0/50751428 shutdown_connections 2024-01-23T08:43:54.247 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.410+0000 7ff314472700 1 -- 172.21.15.59:0/50751428 wait complete. 2024-01-23T08:43:54.247 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.411+0000 7ff314472700 1 Processor -- start 2024-01-23T08:43:54.247 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.411+0000 7ff314472700 1 -- start start 2024-01-23T08:43:54.247 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.411+0000 7ff314472700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff30c102500 0x7ff30c078b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:54.247 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.411+0000 7ff314472700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c079090 con 0x7ff30c102500 2024-01-23T08:43:54.247 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.411+0000 7ff31220e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff30c102500 0x7ff30c078b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:54.247 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.411+0000 7ff31220e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff30c102500 0x7ff30c078b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34528/0 (socket says 172.21.15.59:34528) 2024-01-23T08:43:54.247 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.411+0000 7ff31220e700 1 -- 172.21.15.59:0/2851367415 learned_addr learned my addr 172.21.15.59:0/2851367415 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:54.247 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.411+0000 7ff31220e700 1 -- 172.21.15.59:0/2851367415 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3080094f0 con 0x7ff30c102500 2024-01-23T08:43:54.247 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.412+0000 7ff31220e700 1 --2- 172.21.15.59:0/2851367415 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff30c102500 0x7ff30c078b50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff308003080 tx=0x7ff308009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:54.247 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.412+0000 7ff302ffd700 1 -- 172.21.15.59:0/2851367415 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff308010800 con 0x7ff30c102500 2024-01-23T08:43:54.248 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.412+0000 7ff302ffd700 1 -- 172.21.15.59:0/2851367415 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff308010e10 con 0x7ff30c102500 2024-01-23T08:43:54.248 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.412+0000 7ff314472700 1 -- 172.21.15.59:0/2851367415 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff30c0771b0 con 0x7ff30c102500 2024-01-23T08:43:54.248 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.412+0000 7ff302ffd700 1 -- 172.21.15.59:0/2851367415 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff30801a430 con 0x7ff30c102500 2024-01-23T08:43:54.248 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.412+0000 7ff314472700 1 -- 172.21.15.59:0/2851367415 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff30c077620 con 0x7ff30c102500 2024-01-23T08:43:54.248 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.413+0000 7ff302ffd700 1 -- 172.21.15.59:0/2851367415 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7ff308010960 con 0x7ff30c102500 2024-01-23T08:43:54.248 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.413+0000 7ff314472700 1 -- 172.21.15.59:0/2851367415 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff30c05f450 con 0x7ff30c102500 2024-01-23T08:43:54.248 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.413+0000 7ff302ffd700 1 --2- 172.21.15.59:0/2851367415 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7ff2f80365f0 0x7ff2f8038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:54.248 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.413+0000 7ff302ffd700 1 -- 172.21.15.59:0/2851367415 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff30804a5c0 con 0x7ff30c102500 2024-01-23T08:43:54.248 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.413+0000 7ff311a0d700 1 --2- 172.21.15.59:0/2851367415 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7ff2f80365f0 0x7ff2f8038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:54.248 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.414+0000 7ff311a0d700 1 --2- 172.21.15.59:0/2851367415 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7ff2f80365f0 0x7ff2f8038aa0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc009990 tx=0x7ff2fc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:54.249 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.418+0000 7ff302ffd700 1 -- 172.21.15.59:0/2851367415 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ff308022990 con 0x7ff30c102500 2024-01-23T08:43:54.249 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.607+0000 7ff314472700 1 -- 172.21.15.59:0/2851367415 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7ff30c064940 con 0x7ff30c102500 2024-01-23T08:43:54.249 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.610+0000 7ff302ffd700 1 -- 172.21.15.59:0/2851367415 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7ff3080227b0 con 0x7ff30c102500 2024-01-23T08:43:54.249 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.612+0000 7ff314472700 1 -- 172.21.15.59:0/2851367415 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7ff2f80365f0 msgr2=0x7ff2f8038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:54.249 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.612+0000 7ff314472700 1 --2- 172.21.15.59:0/2851367415 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7ff2f80365f0 0x7ff2f8038aa0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc009990 tx=0x7ff2fc006e40 comp rx=0 tx=0).stop 2024-01-23T08:43:54.249 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.612+0000 7ff314472700 1 -- 172.21.15.59:0/2851367415 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff30c102500 msgr2=0x7ff30c078b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:54.249 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.612+0000 7ff314472700 1 --2- 172.21.15.59:0/2851367415 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff30c102500 0x7ff30c078b50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff308003080 tx=0x7ff308009250 comp rx=0 tx=0).stop 2024-01-23T08:43:54.249 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.612+0000 7ff314472700 1 -- 172.21.15.59:0/2851367415 shutdown_connections 2024-01-23T08:43:54.249 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.612+0000 7ff314472700 1 --2- 172.21.15.59:0/2851367415 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7ff2f80365f0 0x7ff2f8038aa0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:54.249 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.612+0000 7ff314472700 1 --2- 172.21.15.59:0/2851367415 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff30c102500 0x7ff30c078b50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:54.249 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.612+0000 7ff314472700 1 -- 172.21.15.59:0/2851367415 >> 172.21.15.59:0/2851367415 conn(0x7ff30c0fdca0 msgr2=0x7ff30c0feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:54.249 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.612+0000 7ff314472700 1 -- 172.21.15.59:0/2851367415 shutdown_connections 2024-01-23T08:43:54.250 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:53.612+0000 7ff314472700 1 -- 172.21.15.59:0/2851367415 wait complete. 2024-01-23T08:43:54.888 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:54 smithi059 ceph-mon[28424]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:43:55.475 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.692+0000 7f7431f6d700 1 Processor -- start 2024-01-23T08:43:55.475 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.692+0000 7f7431f6d700 1 -- start start 2024-01-23T08:43:55.475 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.692+0000 7f7431f6d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f742c102a70 0x7f742c102e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:55.475 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.692+0000 7f7431f6d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742c103450 con 0x7f742c102a70 2024-01-23T08:43:55.476 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.692+0000 7f742b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f742c102a70 0x7f742c102e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:55.476 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.692+0000 7f742b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f742c102a70 0x7f742c102e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34532/0 (socket says 172.21.15.59:34532) 2024-01-23T08:43:55.476 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.692+0000 7f742b7fe700 1 -- 172.21.15.59:0/3781642083 learned_addr learned my addr 172.21.15.59:0/3781642083 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:55.476 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.693+0000 7f742b7fe700 1 -- 172.21.15.59:0/3781642083 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f742c103590 con 0x7f742c102a70 2024-01-23T08:43:55.476 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.693+0000 7f742b7fe700 1 --2- 172.21.15.59:0/3781642083 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f742c102a70 0x7f742c102e80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f741c0098e0 tx=0x7f741c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6597d1330df14232 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:55.476 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.693+0000 7f742a7fc700 1 -- 172.21.15.59:0/3781642083 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f741c00b860 con 0x7f742c102a70 2024-01-23T08:43:55.476 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.693+0000 7f742a7fc700 1 -- 172.21.15.59:0/3781642083 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f741c00be70 con 0x7f742c102a70 2024-01-23T08:43:55.476 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.693+0000 7f742a7fc700 1 -- 172.21.15.59:0/3781642083 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f741c010380 con 0x7f742c102a70 2024-01-23T08:43:55.476 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.694+0000 7f7431f6d700 1 -- 172.21.15.59:0/3781642083 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f742c102a70 msgr2=0x7f742c102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:55.476 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.694+0000 7f7431f6d700 1 --2- 172.21.15.59:0/3781642083 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f742c102a70 0x7f742c102e80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f741c0098e0 tx=0x7f741c009250 comp rx=0 tx=0).stop 2024-01-23T08:43:55.476 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.694+0000 7f7431f6d700 1 -- 172.21.15.59:0/3781642083 shutdown_connections 2024-01-23T08:43:55.476 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.694+0000 7f7431f6d700 1 --2- 172.21.15.59:0/3781642083 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f742c102a70 0x7f742c102e80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:55.476 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.694+0000 7f7431f6d700 1 -- 172.21.15.59:0/3781642083 >> 172.21.15.59:0/3781642083 conn(0x7f742c0fc260 msgr2=0x7f742c0fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:55.476 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.694+0000 7f7431f6d700 1 -- 172.21.15.59:0/3781642083 shutdown_connections 2024-01-23T08:43:55.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.694+0000 7f7431f6d700 1 -- 172.21.15.59:0/3781642083 wait complete. 2024-01-23T08:43:55.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.694+0000 7f7431f6d700 1 Processor -- start 2024-01-23T08:43:55.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.695+0000 7f7431f6d700 1 -- start start 2024-01-23T08:43:55.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.695+0000 7f7431f6d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f742c102a70 0x7f742c113440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:55.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.695+0000 7f7431f6d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742c113980 con 0x7f742c102a70 2024-01-23T08:43:55.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.695+0000 7f742b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f742c102a70 0x7f742c113440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:55.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.695+0000 7f742b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f742c102a70 0x7f742c113440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34538/0 (socket says 172.21.15.59:34538) 2024-01-23T08:43:55.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.695+0000 7f742b7fe700 1 -- 172.21.15.59:0/2814153184 learned_addr learned my addr 172.21.15.59:0/2814153184 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:55.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.695+0000 7f742b7fe700 1 -- 172.21.15.59:0/2814153184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f741c0094f0 con 0x7f742c102a70 2024-01-23T08:43:55.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.695+0000 7f742b7fe700 1 --2- 172.21.15.59:0/2814153184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f742c102a70 0x7f742c113440 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f741c00c010 tx=0x7f741c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:55.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.695+0000 7f7423fff700 1 -- 172.21.15.59:0/2814153184 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f741c010960 con 0x7f742c102a70 2024-01-23T08:43:55.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.695+0000 7f7431f6d700 1 -- 172.21.15.59:0/2814153184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f742c113b80 con 0x7f742c102a70 2024-01-23T08:43:55.477 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.695+0000 7f7423fff700 1 -- 172.21.15.59:0/2814153184 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f741c01a430 con 0x7f742c102a70 2024-01-23T08:43:55.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.695+0000 7f7423fff700 1 -- 172.21.15.59:0/2814153184 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f741c019580 con 0x7f742c102a70 2024-01-23T08:43:55.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.695+0000 7f7431f6d700 1 -- 172.21.15.59:0/2814153184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f742c113f90 con 0x7f742c102a70 2024-01-23T08:43:55.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.696+0000 7f7423fff700 1 -- 172.21.15.59:0/2814153184 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f741c01a980 con 0x7f742c102a70 2024-01-23T08:43:55.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.696+0000 7f7423fff700 1 --2- 172.21.15.59:0/2814153184 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f74180365f0 0x7f7418038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:55.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.696+0000 7f7431f6d700 1 -- 172.21.15.59:0/2814153184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f742c05f450 con 0x7f742c102a70 2024-01-23T08:43:55.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.697+0000 7f7423fff700 1 -- 172.21.15.59:0/2814153184 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f741c04a4a0 con 0x7f742c102a70 2024-01-23T08:43:55.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.697+0000 7f742affd700 1 --2- 172.21.15.59:0/2814153184 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f74180365f0 0x7f7418038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:55.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.697+0000 7f742affd700 1 --2- 172.21.15.59:0/2814153184 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f74180365f0 0x7f7418038aa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7414009990 tx=0x7f7414006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:55.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.701+0000 7f7423fff700 1 -- 172.21.15.59:0/2814153184 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f741c02bd40 con 0x7f742c102a70 2024-01-23T08:43:55.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.890+0000 7f7431f6d700 1 -- 172.21.15.59:0/2814153184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f742c1169f0 con 0x7f742c102a70 2024-01-23T08:43:55.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.893+0000 7f7423fff700 1 -- 172.21.15.59:0/2814153184 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f741c02b300 con 0x7f742c102a70 2024-01-23T08:43:55.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.895+0000 7f7431f6d700 1 -- 172.21.15.59:0/2814153184 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f74180365f0 msgr2=0x7f7418038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:55.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.895+0000 7f7431f6d700 1 --2- 172.21.15.59:0/2814153184 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f74180365f0 0x7f7418038aa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7414009990 tx=0x7f7414006e40 comp rx=0 tx=0).stop 2024-01-23T08:43:55.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.895+0000 7f7431f6d700 1 -- 172.21.15.59:0/2814153184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f742c102a70 msgr2=0x7f742c113440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:55.478 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.895+0000 7f7431f6d700 1 --2- 172.21.15.59:0/2814153184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f742c102a70 0x7f742c113440 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f741c00c010 tx=0x7f741c009250 comp rx=0 tx=0).stop 2024-01-23T08:43:55.479 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.895+0000 7f7431f6d700 1 -- 172.21.15.59:0/2814153184 shutdown_connections 2024-01-23T08:43:55.479 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.896+0000 7f7431f6d700 1 --2- 172.21.15.59:0/2814153184 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f74180365f0 0x7f7418038aa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:55.479 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.896+0000 7f7431f6d700 1 --2- 172.21.15.59:0/2814153184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f742c102a70 0x7f742c113440 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:55.479 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.896+0000 7f7431f6d700 1 -- 172.21.15.59:0/2814153184 >> 172.21.15.59:0/2814153184 conn(0x7f742c0fc260 msgr2=0x7f742c0fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:55.479 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.896+0000 7f7431f6d700 1 -- 172.21.15.59:0/2814153184 shutdown_connections 2024-01-23T08:43:55.479 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:54.896+0000 7f7431f6d700 1 -- 172.21.15.59:0/2814153184 wait complete. 2024-01-23T08:43:55.479 INFO:teuthology.orchestra.run.smithi059.stdout:Enabling the dashboard module... 2024-01-23T08:43:55.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:55 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/2814153184' entity='client.admin' 2024-01-23T08:43:57.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:56 smithi059 ceph-mon[28424]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:43:57.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:56 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/4157110271' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-01-23T08:43:57.623 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.896+0000 7f50681bc700 1 Processor -- start 2024-01-23T08:43:57.623 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.896+0000 7f50681bc700 1 -- start start 2024-01-23T08:43:57.623 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.896+0000 7f50681bc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5060076ca0 0x7f50601050b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:57.623 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.896+0000 7f50681bc700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5060105680 con 0x7f5060076ca0 2024-01-23T08:43:57.623 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.896+0000 7f5065f58700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5060076ca0 0x7f50601050b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:57.624 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.896+0000 7f5065f58700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5060076ca0 0x7f50601050b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34542/0 (socket says 172.21.15.59:34542) 2024-01-23T08:43:57.624 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.896+0000 7f5065f58700 1 -- 172.21.15.59:0/2544032264 learned_addr learned my addr 172.21.15.59:0/2544032264 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:57.624 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.897+0000 7f5065f58700 1 -- 172.21.15.59:0/2544032264 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50601057c0 con 0x7f5060076ca0 2024-01-23T08:43:57.624 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.897+0000 7f5065f58700 1 --2- 172.21.15.59:0/2544032264 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5060076ca0 0x7f50601050b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f505c0098e0 tx=0x7f505c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2605372328b9836a server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:57.624 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.897+0000 7f5064f56700 1 -- 172.21.15.59:0/2544032264 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f505c00b860 con 0x7f5060076ca0 2024-01-23T08:43:57.624 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.897+0000 7f5064f56700 1 -- 172.21.15.59:0/2544032264 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f505c00be70 con 0x7f5060076ca0 2024-01-23T08:43:57.624 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.897+0000 7f5064f56700 1 -- 172.21.15.59:0/2544032264 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f505c010380 con 0x7f5060076ca0 2024-01-23T08:43:57.624 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.898+0000 7f50681bc700 1 -- 172.21.15.59:0/2544032264 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5060076ca0 msgr2=0x7f50601050b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:57.624 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.898+0000 7f50681bc700 1 --2- 172.21.15.59:0/2544032264 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5060076ca0 0x7f50601050b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f505c0098e0 tx=0x7f505c009250 comp rx=0 tx=0).stop 2024-01-23T08:43:57.624 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.898+0000 7f50681bc700 1 -- 172.21.15.59:0/2544032264 shutdown_connections 2024-01-23T08:43:57.624 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.898+0000 7f50681bc700 1 --2- 172.21.15.59:0/2544032264 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5060076ca0 0x7f50601050b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:57.624 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.898+0000 7f50681bc700 1 -- 172.21.15.59:0/2544032264 >> 172.21.15.59:0/2544032264 conn(0x7f50600fcae0 msgr2=0x7f50600feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:57.624 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.898+0000 7f50681bc700 1 -- 172.21.15.59:0/2544032264 shutdown_connections 2024-01-23T08:43:57.624 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.898+0000 7f50681bc700 1 -- 172.21.15.59:0/2544032264 wait complete. 2024-01-23T08:43:57.625 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.898+0000 7f50681bc700 1 Processor -- start 2024-01-23T08:43:57.625 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.898+0000 7f50681bc700 1 -- start start 2024-01-23T08:43:57.625 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.898+0000 7f50681bc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5060076ca0 0x7f5060111140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:57.625 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.898+0000 7f50681bc700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5060111680 con 0x7f5060076ca0 2024-01-23T08:43:57.625 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.898+0000 7f5065f58700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5060076ca0 0x7f5060111140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:57.625 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.898+0000 7f5065f58700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5060076ca0 0x7f5060111140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34544/0 (socket says 172.21.15.59:34544) 2024-01-23T08:43:57.625 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.899+0000 7f5065f58700 1 -- 172.21.15.59:0/4157110271 learned_addr learned my addr 172.21.15.59:0/4157110271 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:57.625 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.899+0000 7f5065f58700 1 -- 172.21.15.59:0/4157110271 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505c0094f0 con 0x7f5060076ca0 2024-01-23T08:43:57.625 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.899+0000 7f5065f58700 1 --2- 172.21.15.59:0/4157110271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5060076ca0 0x7f5060111140 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f505c003080 tx=0x7f505c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:57.625 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.899+0000 7f50567fc700 1 -- 172.21.15.59:0/4157110271 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f505c010800 con 0x7f5060076ca0 2024-01-23T08:43:57.625 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.899+0000 7f50567fc700 1 -- 172.21.15.59:0/4157110271 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f505c010e10 con 0x7f5060076ca0 2024-01-23T08:43:57.625 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.899+0000 7f50681bc700 1 -- 172.21.15.59:0/4157110271 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5060111880 con 0x7f5060076ca0 2024-01-23T08:43:57.625 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.899+0000 7f50567fc700 1 -- 172.21.15.59:0/4157110271 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f505c01a430 con 0x7f5060076ca0 2024-01-23T08:43:57.625 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.899+0000 7f50681bc700 1 -- 172.21.15.59:0/4157110271 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5060111c90 con 0x7f5060076ca0 2024-01-23T08:43:57.625 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.900+0000 7f50567fc700 1 -- 172.21.15.59:0/4157110271 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f505c01a660 con 0x7f5060076ca0 2024-01-23T08:43:57.625 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.900+0000 7f50567fc700 1 --2- 172.21.15.59:0/4157110271 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f504c036640 0x7f504c038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:57.626 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.900+0000 7f50567fc700 1 -- 172.21.15.59:0/4157110271 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f505c04a580 con 0x7f5060076ca0 2024-01-23T08:43:57.626 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.900+0000 7f5065757700 1 --2- 172.21.15.59:0/4157110271 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f504c036640 0x7f504c038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:57.626 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.900+0000 7f50681bc700 1 -- 172.21.15.59:0/4157110271 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f506005f450 con 0x7f5060076ca0 2024-01-23T08:43:57.626 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.901+0000 7f5065757700 1 --2- 172.21.15.59:0/4157110271 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f504c036640 0x7f504c038af0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5050009990 tx=0x7f5050006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:57.626 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:55.905+0000 7f50567fc700 1 -- 172.21.15.59:0/4157110271 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f505c01a960 con 0x7f5060076ca0 2024-01-23T08:43:57.626 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:56.152+0000 7f50681bc700 1 -- 172.21.15.59:0/4157110271 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f50601065a0 con 0x7f5060076ca0 2024-01-23T08:43:57.626 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:56.976+0000 7f50567fc700 1 -- 172.21.15.59:0/4157110271 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 9) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f505c00fdc0 con 0x7f5060076ca0 2024-01-23T08:43:57.626 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:56.978+0000 7f50567fc700 1 -- 172.21.15.59:0/4157110271 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f505c024790 con 0x7f5060076ca0 2024-01-23T08:43:57.626 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:56.981+0000 7f50681bc700 1 -- 172.21.15.59:0/4157110271 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f504c036640 msgr2=0x7f504c038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:57.626 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:56.981+0000 7f50681bc700 1 --2- 172.21.15.59:0/4157110271 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f504c036640 0x7f504c038af0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5050009990 tx=0x7f5050006e40 comp rx=0 tx=0).stop 2024-01-23T08:43:57.626 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:56.981+0000 7f50681bc700 1 -- 172.21.15.59:0/4157110271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5060076ca0 msgr2=0x7f5060111140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:57.626 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:56.981+0000 7f50681bc700 1 --2- 172.21.15.59:0/4157110271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5060076ca0 0x7f5060111140 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f505c003080 tx=0x7f505c009250 comp rx=0 tx=0).stop 2024-01-23T08:43:57.626 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:56.981+0000 7f50681bc700 1 -- 172.21.15.59:0/4157110271 shutdown_connections 2024-01-23T08:43:57.626 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:56.981+0000 7f50681bc700 1 --2- 172.21.15.59:0/4157110271 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7f504c036640 0x7f504c038af0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:57.627 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:56.981+0000 7f50681bc700 1 --2- 172.21.15.59:0/4157110271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5060076ca0 0x7f5060111140 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:57.627 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:56.981+0000 7f50681bc700 1 -- 172.21.15.59:0/4157110271 >> 172.21.15.59:0/4157110271 conn(0x7f50600fcae0 msgr2=0x7f50600fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:57.627 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:56.982+0000 7f50681bc700 1 -- 172.21.15.59:0/4157110271 shutdown_connections 2024-01-23T08:43:57.627 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:56.982+0000 7f50681bc700 1 -- 172.21.15.59:0/4157110271 wait complete. 2024-01-23T08:43:58.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:57 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/4157110271' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-01-23T08:43:58.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:57 smithi059 ceph-mon[28424]: mgrmap e9: smithi059.bzqirf(active, since 25s) 2024-01-23T08:43:58.865 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout { 2024-01-23T08:43:58.865 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-01-23T08:43:58.865 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "available": true, 2024-01-23T08:43:58.865 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "active_name": "smithi059.bzqirf", 2024-01-23T08:43:58.865 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-01-23T08:43:58.865 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout } 2024-01-23T08:43:58.865 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.026+0000 7fa8ea529700 1 Processor -- start 2024-01-23T08:43:58.865 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.026+0000 7fa8ea529700 1 -- start start 2024-01-23T08:43:58.865 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.026+0000 7fa8ea529700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8e4102500 0x7fa8e4102910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:58.865 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.026+0000 7fa8ea529700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e4102e50 con 0x7fa8e4102500 2024-01-23T08:43:58.866 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.026+0000 7fa8e3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8e4102500 0x7fa8e4102910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:58.866 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.026+0000 7fa8e3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8e4102500 0x7fa8e4102910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32860/0 (socket says 172.21.15.59:32860) 2024-01-23T08:43:58.866 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.026+0000 7fa8e3fff700 1 -- 172.21.15.59:0/2460873502 learned_addr learned my addr 172.21.15.59:0/2460873502 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:58.866 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.026+0000 7fa8e3fff700 1 -- 172.21.15.59:0/2460873502 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e4102f90 con 0x7fa8e4102500 2024-01-23T08:43:58.866 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.027+0000 7fa8e3fff700 1 --2- 172.21.15.59:0/2460873502 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8e4102500 0x7fa8e4102910 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa8d40098e0 tx=0x7fa8d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e1bb078bf5109a4 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:58.866 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.027+0000 7fa8e2ffd700 1 -- 172.21.15.59:0/2460873502 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8d400b860 con 0x7fa8e4102500 2024-01-23T08:43:58.866 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.027+0000 7fa8e2ffd700 1 -- 172.21.15.59:0/2460873502 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa8d400be70 con 0x7fa8e4102500 2024-01-23T08:43:58.866 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.027+0000 7fa8e2ffd700 1 -- 172.21.15.59:0/2460873502 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8d4010380 con 0x7fa8e4102500 2024-01-23T08:43:58.866 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.027+0000 7fa8ea529700 1 -- 172.21.15.59:0/2460873502 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8e4102500 msgr2=0x7fa8e4102910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:58.866 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.027+0000 7fa8ea529700 1 --2- 172.21.15.59:0/2460873502 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8e4102500 0x7fa8e4102910 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa8d40098e0 tx=0x7fa8d4009250 comp rx=0 tx=0).stop 2024-01-23T08:43:58.866 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.027+0000 7fa8ea529700 1 -- 172.21.15.59:0/2460873502 shutdown_connections 2024-01-23T08:43:58.866 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.027+0000 7fa8ea529700 1 --2- 172.21.15.59:0/2460873502 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8e4102500 0x7fa8e4102910 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:58.866 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.027+0000 7fa8ea529700 1 -- 172.21.15.59:0/2460873502 >> 172.21.15.59:0/2460873502 conn(0x7fa8e40fdc80 msgr2=0x7fa8e4100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:58.866 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.027+0000 7fa8ea529700 1 -- 172.21.15.59:0/2460873502 shutdown_connections 2024-01-23T08:43:58.866 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.027+0000 7fa8ea529700 1 -- 172.21.15.59:0/2460873502 wait complete. 2024-01-23T08:43:58.867 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.028+0000 7fa8ea529700 1 Processor -- start 2024-01-23T08:43:58.867 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.028+0000 7fa8ea529700 1 -- start start 2024-01-23T08:43:58.867 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.028+0000 7fa8ea529700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8e4102500 0x7fa8e41114a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:58.867 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.028+0000 7fa8ea529700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e41119e0 con 0x7fa8e4102500 2024-01-23T08:43:58.867 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.028+0000 7fa8e3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8e4102500 0x7fa8e41114a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:43:58.867 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.028+0000 7fa8e3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8e4102500 0x7fa8e41114a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32872/0 (socket says 172.21.15.59:32872) 2024-01-23T08:43:58.867 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.028+0000 7fa8e3fff700 1 -- 172.21.15.59:0/55512268 learned_addr learned my addr 172.21.15.59:0/55512268 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:43:58.867 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.029+0000 7fa8e3fff700 1 -- 172.21.15.59:0/55512268 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d40094f0 con 0x7fa8e4102500 2024-01-23T08:43:58.867 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.029+0000 7fa8e3fff700 1 --2- 172.21.15.59:0/55512268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8e4102500 0x7fa8e41114a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa8d400bac0 tx=0x7fa8d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:43:58.867 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.029+0000 7fa8e0ff9700 1 -- 172.21.15.59:0/55512268 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8d40109b0 con 0x7fa8e4102500 2024-01-23T08:43:58.867 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.029+0000 7fa8ea529700 1 -- 172.21.15.59:0/55512268 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e4111be0 con 0x7fa8e4102500 2024-01-23T08:43:58.867 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.029+0000 7fa8e0ff9700 1 -- 172.21.15.59:0/55512268 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa8d401a430 con 0x7fa8e4102500 2024-01-23T08:43:58.867 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.029+0000 7fa8e0ff9700 1 -- 172.21.15.59:0/55512268 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8d40245b0 con 0x7fa8e4102500 2024-01-23T08:43:58.867 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.029+0000 7fa8ea529700 1 -- 172.21.15.59:0/55512268 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e4111ff0 con 0x7fa8e4102500 2024-01-23T08:43:58.867 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.030+0000 7fa8ea529700 1 -- 172.21.15.59:0/55512268 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e410b3c0 con 0x7fa8e4102500 2024-01-23T08:43:58.868 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.030+0000 7fa8e0ff9700 1 -- 172.21.15.59:0/55512268 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 9) v1 ==== 43294+0+0 (secure 0 0 0) 0x7fa8d4010b10 con 0x7fa8e4102500 2024-01-23T08:43:58.868 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.030+0000 7fa8e0ff9700 1 --2- 172.21.15.59:0/55512268 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fa8d0036640 0x7fa8d0038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:43:58.868 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.030+0000 7fa8e0ff9700 1 -- 172.21.15.59:0/55512268 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa8d404a5c0 con 0x7fa8e4102500 2024-01-23T08:43:58.868 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.030+0000 7fa8e37fe700 1 -- 172.21.15.59:0/55512268 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fa8d0036640 msgr2=0x7fa8d0038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1337467262 2024-01-23T08:43:58.868 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.030+0000 7fa8e37fe700 1 --2- 172.21.15.59:0/55512268 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fa8d0036640 0x7fa8d0038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-23T08:43:58.868 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.035+0000 7fa8e0ff9700 1 -- 172.21.15.59:0/55512268 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa8d401a5a0 con 0x7fa8e4102500 2024-01-23T08:43:58.868 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.232+0000 7fa8e37fe700 1 -- 172.21.15.59:0/55512268 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fa8d0036640 msgr2=0x7fa8d0038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1337467262 2024-01-23T08:43:58.868 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.232+0000 7fa8e37fe700 1 --2- 172.21.15.59:0/55512268 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fa8d0036640 0x7fa8d0038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-23T08:43:58.868 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.282+0000 7fa8ea529700 1 -- 172.21.15.59:0/55512268 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fa8e4064990 con 0x7fa8e4102500 2024-01-23T08:43:58.868 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.282+0000 7fa8e0ff9700 1 -- 172.21.15.59:0/55512268 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7fa8d4018b40 con 0x7fa8e4102500 2024-01-23T08:43:58.868 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.284+0000 7fa8ea529700 1 -- 172.21.15.59:0/55512268 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fa8d0036640 msgr2=0x7fa8d0038af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-23T08:43:58.868 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.284+0000 7fa8ea529700 1 --2- 172.21.15.59:0/55512268 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fa8d0036640 0x7fa8d0038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:58.868 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.284+0000 7fa8ea529700 1 -- 172.21.15.59:0/55512268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8e4102500 msgr2=0x7fa8e41114a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:43:58.869 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.284+0000 7fa8ea529700 1 --2- 172.21.15.59:0/55512268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8e4102500 0x7fa8e41114a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa8d400bac0 tx=0x7fa8d4009250 comp rx=0 tx=0).stop 2024-01-23T08:43:58.869 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.285+0000 7fa8ea529700 1 -- 172.21.15.59:0/55512268 shutdown_connections 2024-01-23T08:43:58.869 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.285+0000 7fa8ea529700 1 --2- 172.21.15.59:0/55512268 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fa8d0036640 0x7fa8d0038af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:58.869 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.285+0000 7fa8ea529700 1 --2- 172.21.15.59:0/55512268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8e4102500 0x7fa8e41114a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:43:58.869 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.285+0000 7fa8ea529700 1 -- 172.21.15.59:0/55512268 >> 172.21.15.59:0/55512268 conn(0x7fa8e40fdc80 msgr2=0x7fa8e40feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:43:58.869 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.285+0000 7fa8ea529700 1 -- 172.21.15.59:0/55512268 shutdown_connections 2024-01-23T08:43:58.869 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:58.285+0000 7fa8ea529700 1 -- 172.21.15.59:0/55512268 wait complete. 2024-01-23T08:43:58.869 INFO:teuthology.orchestra.run.smithi059.stdout:Waiting for the mgr to restart... 2024-01-23T08:43:58.869 INFO:teuthology.orchestra.run.smithi059.stdout:Waiting for mgr epoch 9... 2024-01-23T08:43:59.143 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:43:58 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/55512268' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-23T08:44:04.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:03 smithi059 ceph-mon[28424]: Active manager daemon smithi059.bzqirf restarted 2024-01-23T08:44:04.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:03 smithi059 ceph-mon[28424]: Activating manager daemon smithi059.bzqirf 2024-01-23T08:44:04.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:03 smithi059 ceph-mon[28424]: osdmap e3: 0 total, 0 up, 0 in 2024-01-23T08:44:04.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:03 smithi059 ceph-mon[28424]: mgrmap e10: smithi059.bzqirf(active, starting, since 0.00388328s) 2024-01-23T08:44:04.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:03 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-01-23T08:44:04.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:03 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mgr metadata", "who": "smithi059.bzqirf", "id": "smithi059.bzqirf"}]: dispatch 2024-01-23T08:44:04.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:03 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-23T08:44:04.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:03 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-23T08:44:04.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:03 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-23T08:44:04.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:03 smithi059 ceph-mon[28424]: Manager daemon smithi059.bzqirf is now available 2024-01-23T08:44:04.223 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:03 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:44:04.223 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:03 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:44:04.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:04 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.bzqirf/mirror_snapshot_schedule"}]: dispatch 2024-01-23T08:44:04.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:04 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.bzqirf/trash_purge_schedule"}]: dispatch 2024-01-23T08:44:04.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:04 smithi059 ceph-mon[28424]: mgrmap e11: smithi059.bzqirf(active, since 1.00533s) 2024-01-23T08:44:05.491 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout { 2024-01-23T08:44:05.491 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-01-23T08:44:05.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "initialized": true 2024-01-23T08:44:05.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout } 2024-01-23T08:44:05.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.268+0000 7fc2c3ed9700 1 Processor -- start 2024-01-23T08:44:05.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.268+0000 7fc2c3ed9700 1 -- start start 2024-01-23T08:44:05.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.268+0000 7fc2c3ed9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2bc078d90 0x7fc2bc077230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:05.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.268+0000 7fc2c3ed9700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc077770 con 0x7fc2bc078d90 2024-01-23T08:44:05.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.268+0000 7fc2c1c75700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2bc078d90 0x7fc2bc077230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:05.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.268+0000 7fc2c1c75700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2bc078d90 0x7fc2bc077230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32878/0 (socket says 172.21.15.59:32878) 2024-01-23T08:44:05.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.268+0000 7fc2c1c75700 1 -- 172.21.15.59:0/2269923735 learned_addr learned my addr 172.21.15.59:0/2269923735 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:44:05.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.269+0000 7fc2c1c75700 1 -- 172.21.15.59:0/2269923735 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2bc0778b0 con 0x7fc2bc078d90 2024-01-23T08:44:05.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.269+0000 7fc2c1c75700 1 --2- 172.21.15.59:0/2269923735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2bc078d90 0x7fc2bc077230 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc2b80098e0 tx=0x7fc2b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62095ae8203aabde server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:05.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.269+0000 7fc2c0c73700 1 -- 172.21.15.59:0/2269923735 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2b800b860 con 0x7fc2bc078d90 2024-01-23T08:44:05.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.269+0000 7fc2c0c73700 1 -- 172.21.15.59:0/2269923735 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc2b800be70 con 0x7fc2bc078d90 2024-01-23T08:44:05.492 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.269+0000 7fc2c0c73700 1 -- 172.21.15.59:0/2269923735 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2b8010380 con 0x7fc2bc078d90 2024-01-23T08:44:05.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.269+0000 7fc2c3ed9700 1 -- 172.21.15.59:0/2269923735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2bc078d90 msgr2=0x7fc2bc077230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:05.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.269+0000 7fc2c3ed9700 1 --2- 172.21.15.59:0/2269923735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2bc078d90 0x7fc2bc077230 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc2b80098e0 tx=0x7fc2b8009250 comp rx=0 tx=0).stop 2024-01-23T08:44:05.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.270+0000 7fc2c3ed9700 1 -- 172.21.15.59:0/2269923735 shutdown_connections 2024-01-23T08:44:05.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.270+0000 7fc2c3ed9700 1 --2- 172.21.15.59:0/2269923735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2bc078d90 0x7fc2bc077230 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:05.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.270+0000 7fc2c3ed9700 1 -- 172.21.15.59:0/2269923735 >> 172.21.15.59:0/2269923735 conn(0x7fc2bc0fdbd0 msgr2=0x7fc2bc0fffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:05.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.270+0000 7fc2c3ed9700 1 -- 172.21.15.59:0/2269923735 shutdown_connections 2024-01-23T08:44:05.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.270+0000 7fc2c3ed9700 1 -- 172.21.15.59:0/2269923735 wait complete. 2024-01-23T08:44:05.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.270+0000 7fc2c3ed9700 1 Processor -- start 2024-01-23T08:44:05.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.270+0000 7fc2c3ed9700 1 -- start start 2024-01-23T08:44:05.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.270+0000 7fc2c3ed9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2bc078d90 0x7fc2bc1133b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:05.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.270+0000 7fc2c3ed9700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc1138f0 con 0x7fc2bc078d90 2024-01-23T08:44:05.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.270+0000 7fc2c1c75700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2bc078d90 0x7fc2bc1133b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:05.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.270+0000 7fc2c1c75700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2bc078d90 0x7fc2bc1133b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32892/0 (socket says 172.21.15.59:32892) 2024-01-23T08:44:05.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.271+0000 7fc2c1c75700 1 -- 172.21.15.59:0/1308239077 learned_addr learned my addr 172.21.15.59:0/1308239077 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:44:05.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.271+0000 7fc2c1c75700 1 -- 172.21.15.59:0/1308239077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b80094f0 con 0x7fc2bc078d90 2024-01-23T08:44:05.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.271+0000 7fc2c1c75700 1 --2- 172.21.15.59:0/1308239077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2bc078d90 0x7fc2bc1133b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc2b80094c0 tx=0x7fc2b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:05.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.271+0000 7fc2b27fc700 1 -- 172.21.15.59:0/1308239077 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2b8010850 con 0x7fc2bc078d90 2024-01-23T08:44:05.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.271+0000 7fc2b27fc700 1 -- 172.21.15.59:0/1308239077 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc2b8010e60 con 0x7fc2bc078d90 2024-01-23T08:44:05.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.271+0000 7fc2c3ed9700 1 -- 172.21.15.59:0/1308239077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2bc113af0 con 0x7fc2bc078d90 2024-01-23T08:44:05.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.271+0000 7fc2b27fc700 1 -- 172.21.15.59:0/1308239077 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2b80243f0 con 0x7fc2bc078d90 2024-01-23T08:44:05.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.271+0000 7fc2c3ed9700 1 -- 172.21.15.59:0/1308239077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2bc113f00 con 0x7fc2bc078d90 2024-01-23T08:44:05.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.272+0000 7fc2b27fc700 1 -- 172.21.15.59:0/1308239077 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 9) v1 ==== 43294+0+0 (secure 0 0 0) 0x7fc2b80246b0 con 0x7fc2bc078d90 2024-01-23T08:44:05.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.272+0000 7fc2b27fc700 1 --2- 172.21.15.59:0/1308239077 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc2a8036640 0x7fc2a8038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:05.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.272+0000 7fc2c3ed9700 1 -- 172.21.15.59:0/1308239077 --> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc2bc05f450 con 0x7fc2a8036640 2024-01-23T08:44:05.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.272+0000 7fc2b27fc700 1 -- 172.21.15.59:0/1308239077 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fc2b804a680 con 0x7fc2bc078d90 2024-01-23T08:44:05.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.272+0000 7fc2c1474700 1 -- 172.21.15.59:0/1308239077 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc2a8036640 msgr2=0x7fc2a8038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1337467262 2024-01-23T08:44:05.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.272+0000 7fc2c1474700 1 --2- 172.21.15.59:0/1308239077 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc2a8036640 0x7fc2a8038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-23T08:44:05.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.473+0000 7fc2c1474700 1 -- 172.21.15.59:0/1308239077 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc2a8036640 msgr2=0x7fc2a8038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1337467262 2024-01-23T08:44:05.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.473+0000 7fc2c1474700 1 --2- 172.21.15.59:0/1308239077 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc2a8036640 0x7fc2a8038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-23T08:44:05.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.874+0000 7fc2c1474700 1 -- 172.21.15.59:0/1308239077 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc2a8036640 msgr2=0x7fc2a8038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1337467262 2024-01-23T08:44:05.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:43:59.874+0000 7fc2c1474700 1 --2- 172.21.15.59:0/1308239077 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc2a8036640 0x7fc2a8038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-23T08:44:05.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:00.675+0000 7fc2c1474700 1 -- 172.21.15.59:0/1308239077 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc2a8036640 msgr2=0x7fc2a8038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1337467262 2024-01-23T08:44:05.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:00.675+0000 7fc2c1474700 1 --2- 172.21.15.59:0/1308239077 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc2a8036640 0x7fc2a8038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-23T08:44:05.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:02.276+0000 7fc2c1474700 1 -- 172.21.15.59:0/1308239077 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc2a8036640 msgr2=0x7fc2a8038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1337467262 2024-01-23T08:44:05.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:02.276+0000 7fc2c1474700 1 --2- 172.21.15.59:0/1308239077 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc2a8036640 0x7fc2a8038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-23T08:44:05.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:03.690+0000 7fc2b27fc700 1 -- 172.21.15.59:0/1308239077 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mgrmap(e 10) v1 ==== 43061+0+0 (secure 0 0 0) 0x7fc2b800fdc0 con 0x7fc2bc078d90 2024-01-23T08:44:05.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:03.690+0000 7fc2b27fc700 1 -- 172.21.15.59:0/1308239077 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc2a8036640 msgr2=0x7fc2a8038af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-23T08:44:05.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:03.690+0000 7fc2b27fc700 1 --2- 172.21.15.59:0/1308239077 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc2a8036640 0x7fc2a8038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:05.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:04.691+0000 7fc2b27fc700 1 -- 172.21.15.59:0/1308239077 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 11) v1 ==== 43188+0+0 (secure 0 0 0) 0x7fc2b802e7c0 con 0x7fc2bc078d90 2024-01-23T08:44:05.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:04.691+0000 7fc2b27fc700 1 --2- 172.21.15.59:0/1308239077 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fc2a8039f40 0x7fc2a803c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:05.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:04.691+0000 7fc2b27fc700 1 -- 172.21.15.59:0/1308239077 --> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc2bc05f450 con 0x7fc2a8039f40 2024-01-23T08:44:05.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:04.692+0000 7fc2c1474700 1 --2- 172.21.15.59:0/1308239077 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fc2a8039f40 0x7fc2a803c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:05.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:04.693+0000 7fc2c1474700 1 --2- 172.21.15.59:0/1308239077 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fc2a8039f40 0x7fc2a803c320 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0039d0 tx=0x7fc2ac0092f0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:05.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:04.693+0000 7fc2b27fc700 1 -- 172.21.15.59:0/1308239077 <== mgr.14162 v2:172.21.15.59:6800/3614124404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fc2bc05f450 con 0x7fc2a8039f40 2024-01-23T08:44:05.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:04.699+0000 7fc2c3ed9700 1 -- 172.21.15.59:0/1308239077 --> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fc2bc108750 con 0x7fc2a8039f40 2024-01-23T08:44:05.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:04.699+0000 7fc2b27fc700 1 -- 172.21.15.59:0/1308239077 <== mgr.14162 v2:172.21.15.59:6800/3614124404 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fc2bc108750 con 0x7fc2a8039f40 2024-01-23T08:44:05.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:04.700+0000 7fc2c3ed9700 1 -- 172.21.15.59:0/1308239077 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fc2a8039f40 msgr2=0x7fc2a803c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:05.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:04.700+0000 7fc2c3ed9700 1 --2- 172.21.15.59:0/1308239077 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fc2a8039f40 0x7fc2a803c320 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0039d0 tx=0x7fc2ac0092f0 comp rx=0 tx=0).stop 2024-01-23T08:44:05.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:04.700+0000 7fc2c3ed9700 1 -- 172.21.15.59:0/1308239077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2bc078d90 msgr2=0x7fc2bc1133b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:05.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:04.700+0000 7fc2c3ed9700 1 --2- 172.21.15.59:0/1308239077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2bc078d90 0x7fc2bc1133b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc2b80094c0 tx=0x7fc2b8009250 comp rx=0 tx=0).stop 2024-01-23T08:44:05.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:04.700+0000 7fc2c3ed9700 1 -- 172.21.15.59:0/1308239077 shutdown_connections 2024-01-23T08:44:05.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:04.700+0000 7fc2c3ed9700 1 --2- 172.21.15.59:0/1308239077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2bc078d90 0x7fc2bc1133b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:05.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:04.700+0000 7fc2c3ed9700 1 --2- 172.21.15.59:0/1308239077 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fc2a8039f40 0x7fc2a803c320 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:05.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:04.700+0000 7fc2c3ed9700 1 --2- 172.21.15.59:0/1308239077 >> [v2:172.21.15.59:6800/1337467262,v1:172.21.15.59:6801/1337467262] conn(0x7fc2a8036640 0x7fc2a8038af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:05.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:04.700+0000 7fc2c3ed9700 1 -- 172.21.15.59:0/1308239077 >> 172.21.15.59:0/1308239077 conn(0x7fc2bc0fdbd0 msgr2=0x7fc2bc0ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:05.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:04.700+0000 7fc2c3ed9700 1 -- 172.21.15.59:0/1308239077 shutdown_connections 2024-01-23T08:44:05.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:04.700+0000 7fc2c3ed9700 1 -- 172.21.15.59:0/1308239077 wait complete. 2024-01-23T08:44:05.496 INFO:teuthology.orchestra.run.smithi059.stdout:mgr epoch 9 is available 2024-01-23T08:44:05.497 INFO:teuthology.orchestra.run.smithi059.stdout:Generating a dashboard self-signed certificate... 2024-01-23T08:44:06.190 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:05 smithi059 ceph-mon[28424]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-23T08:44:06.190 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:05 smithi059 ceph-mon[28424]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-23T08:44:06.190 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:05 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:06.190 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:05 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:06.796 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-01-23T08:44:06.796 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.985+0000 7fcf3c8bc700 1 Processor -- start 2024-01-23T08:44:06.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.985+0000 7fcf3c8bc700 1 -- start start 2024-01-23T08:44:06.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.985+0000 7fcf3c8bc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf38102500 0x7fcf38102910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:06.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.985+0000 7fcf3c8bc700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf38102e50 con 0x7fcf38102500 2024-01-23T08:44:06.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.985+0000 7fcf3659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf38102500 0x7fcf38102910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:06.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.985+0000 7fcf3659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf38102500 0x7fcf38102910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32976/0 (socket says 172.21.15.59:32976) 2024-01-23T08:44:06.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.985+0000 7fcf3659c700 1 -- 172.21.15.59:0/4212526494 learned_addr learned my addr 172.21.15.59:0/4212526494 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:44:06.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.985+0000 7fcf3659c700 1 -- 172.21.15.59:0/4212526494 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf38102f90 con 0x7fcf38102500 2024-01-23T08:44:06.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.986+0000 7fcf3659c700 1 --2- 172.21.15.59:0/4212526494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf38102500 0x7fcf38102910 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcf30009e50 tx=0x7fcf30009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5efe9a8d3eb07d71 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:06.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.986+0000 7fcf3559a700 1 -- 172.21.15.59:0/4212526494 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf300092e0 con 0x7fcf38102500 2024-01-23T08:44:06.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.986+0000 7fcf3559a700 1 -- 172.21.15.59:0/4212526494 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf30012430 con 0x7fcf38102500 2024-01-23T08:44:06.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.986+0000 7fcf3559a700 1 -- 172.21.15.59:0/4212526494 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf300075c0 con 0x7fcf38102500 2024-01-23T08:44:06.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.986+0000 7fcf3c8bc700 1 -- 172.21.15.59:0/4212526494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf38102500 msgr2=0x7fcf38102910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:06.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.986+0000 7fcf3c8bc700 1 --2- 172.21.15.59:0/4212526494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf38102500 0x7fcf38102910 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcf30009e50 tx=0x7fcf30009840 comp rx=0 tx=0).stop 2024-01-23T08:44:06.797 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.986+0000 7fcf3c8bc700 1 -- 172.21.15.59:0/4212526494 shutdown_connections 2024-01-23T08:44:06.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.986+0000 7fcf3c8bc700 1 --2- 172.21.15.59:0/4212526494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf38102500 0x7fcf38102910 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:06.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.986+0000 7fcf3c8bc700 1 -- 172.21.15.59:0/4212526494 >> 172.21.15.59:0/4212526494 conn(0x7fcf380fdc80 msgr2=0x7fcf38100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:06.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.986+0000 7fcf3c8bc700 1 -- 172.21.15.59:0/4212526494 shutdown_connections 2024-01-23T08:44:06.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.987+0000 7fcf3c8bc700 1 -- 172.21.15.59:0/4212526494 wait complete. 2024-01-23T08:44:06.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.987+0000 7fcf3c8bc700 1 Processor -- start 2024-01-23T08:44:06.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.987+0000 7fcf3c8bc700 1 -- start start 2024-01-23T08:44:06.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.987+0000 7fcf3c8bc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf38102500 0x7fcf38111510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:06.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.987+0000 7fcf3c8bc700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf38111a50 con 0x7fcf38102500 2024-01-23T08:44:06.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.987+0000 7fcf3659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf38102500 0x7fcf38111510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:06.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.987+0000 7fcf3659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf38102500 0x7fcf38111510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32982/0 (socket says 172.21.15.59:32982) 2024-01-23T08:44:06.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.987+0000 7fcf3659c700 1 -- 172.21.15.59:0/1571018336 learned_addr learned my addr 172.21.15.59:0/1571018336 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:44:06.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.988+0000 7fcf3659c700 1 -- 172.21.15.59:0/1571018336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf30009ae0 con 0x7fcf38102500 2024-01-23T08:44:06.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.988+0000 7fcf3659c700 1 --2- 172.21.15.59:0/1571018336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf38102500 0x7fcf38111510 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcf3000bc80 tx=0x7fcf30009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:06.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.988+0000 7fcf2effd700 1 -- 172.21.15.59:0/1571018336 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf300092e0 con 0x7fcf38102500 2024-01-23T08:44:06.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.988+0000 7fcf2effd700 1 -- 172.21.15.59:0/1571018336 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf30012900 con 0x7fcf38102500 2024-01-23T08:44:06.798 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.988+0000 7fcf3c8bc700 1 -- 172.21.15.59:0/1571018336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf38111c50 con 0x7fcf38102500 2024-01-23T08:44:06.799 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.988+0000 7fcf2effd700 1 -- 172.21.15.59:0/1571018336 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf30025970 con 0x7fcf38102500 2024-01-23T08:44:06.799 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.988+0000 7fcf3c8bc700 1 -- 172.21.15.59:0/1571018336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf38112060 con 0x7fcf38102500 2024-01-23T08:44:06.799 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.989+0000 7fcf3c8bc700 1 -- 172.21.15.59:0/1571018336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf3810b3c0 con 0x7fcf38102500 2024-01-23T08:44:06.799 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.989+0000 7fcf2effd700 1 -- 172.21.15.59:0/1571018336 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 11) v1 ==== 43188+0+0 (secure 0 0 0) 0x7fcf30019050 con 0x7fcf38102500 2024-01-23T08:44:06.799 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.989+0000 7fcf2effd700 1 --2- 172.21.15.59:0/1571018336 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fcf24036570 0x7fcf24038a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:06.799 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.989+0000 7fcf2effd700 1 -- 172.21.15.59:0/1571018336 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fcf30049e80 con 0x7fcf38102500 2024-01-23T08:44:06.799 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.989+0000 7fcf35d9b700 1 --2- 172.21.15.59:0/1571018336 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fcf24036570 0x7fcf24038a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:06.799 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.989+0000 7fcf35d9b700 1 --2- 172.21.15.59:0/1571018336 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fcf24036570 0x7fcf24038a20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcf20009990 tx=0x7fcf20006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:06.799 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:05.994+0000 7fcf2effd700 1 -- 172.21.15.59:0/1571018336 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fcf30019710 con 0x7fcf38102500 2024-01-23T08:44:06.799 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:06.192+0000 7fcf3c8bc700 1 -- 172.21.15.59:0/1571018336 --> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf3805e260 con 0x7fcf24036570 2024-01-23T08:44:06.799 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:06.250+0000 7fcf2effd700 1 -- 172.21.15.59:0/1571018336 <== mgr.14162 v2:172.21.15.59:6800/3614124404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fcf3805e260 con 0x7fcf24036570 2024-01-23T08:44:06.799 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:06.252+0000 7fcf3c8bc700 1 -- 172.21.15.59:0/1571018336 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fcf24036570 msgr2=0x7fcf24038a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:06.799 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:06.252+0000 7fcf3c8bc700 1 --2- 172.21.15.59:0/1571018336 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fcf24036570 0x7fcf24038a20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcf20009990 tx=0x7fcf20006e40 comp rx=0 tx=0).stop 2024-01-23T08:44:06.799 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:06.252+0000 7fcf3c8bc700 1 -- 172.21.15.59:0/1571018336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf38102500 msgr2=0x7fcf38111510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:06.799 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:06.252+0000 7fcf3c8bc700 1 --2- 172.21.15.59:0/1571018336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf38102500 0x7fcf38111510 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcf3000bc80 tx=0x7fcf30009840 comp rx=0 tx=0).stop 2024-01-23T08:44:06.800 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:06.252+0000 7fcf3c8bc700 1 -- 172.21.15.59:0/1571018336 shutdown_connections 2024-01-23T08:44:06.800 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:06.252+0000 7fcf3c8bc700 1 --2- 172.21.15.59:0/1571018336 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fcf24036570 0x7fcf24038a20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:06.800 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:06.252+0000 7fcf3c8bc700 1 --2- 172.21.15.59:0/1571018336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf38102500 0x7fcf38111510 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:06.800 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:06.252+0000 7fcf3c8bc700 1 -- 172.21.15.59:0/1571018336 >> 172.21.15.59:0/1571018336 conn(0x7fcf380fdc80 msgr2=0x7fcf380fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:06.800 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:06.253+0000 7fcf3c8bc700 1 -- 172.21.15.59:0/1571018336 shutdown_connections 2024-01-23T08:44:06.800 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:06.253+0000 7fcf3c8bc700 1 -- 172.21.15.59:0/1571018336 wait complete. 2024-01-23T08:44:06.800 INFO:teuthology.orchestra.run.smithi059.stdout:Creating initial admin user... 2024-01-23T08:44:07.192 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:06 smithi059 ceph-mon[28424]: [23/Jan/2024:08:44:05] ENGINE Bus STARTING 2024-01-23T08:44:07.192 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:06 smithi059 ceph-mon[28424]: [23/Jan/2024:08:44:05] ENGINE Serving on https://172.21.15.59:7150 2024-01-23T08:44:07.192 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:06 smithi059 ceph-mon[28424]: [23/Jan/2024:08:44:05] ENGINE Bus STARTED 2024-01-23T08:44:07.192 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:06 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:07.192 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:06 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:07.192 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:06 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:07.193 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:06 smithi059 ceph-mon[28424]: mgrmap e12: smithi059.bzqirf(active, since 2s) 2024-01-23T08:44:08.124 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:07 smithi059 ceph-mon[28424]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:44:08.124 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:07 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:08.390 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$f9yw65aBCfSNGdob3PBlbOuv..rbiZemwXhhNQHX9fOe6xx5jn1Ze", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1705999447, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-01-23T08:44:08.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.219+0000 7f43c9381700 1 Processor -- start 2024-01-23T08:44:08.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.219+0000 7f43c9381700 1 -- start start 2024-01-23T08:44:08.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.220+0000 7f43c9381700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43c4102500 0x7f43c4102910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:08.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.220+0000 7f43c9381700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c4102e50 con 0x7f43c4102500 2024-01-23T08:44:08.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.220+0000 7f43c2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43c4102500 0x7f43c4102910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:08.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.220+0000 7f43c2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43c4102500 0x7f43c4102910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32990/0 (socket says 172.21.15.59:32990) 2024-01-23T08:44:08.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.220+0000 7f43c2ffd700 1 -- 172.21.15.59:0/1565340243 learned_addr learned my addr 172.21.15.59:0/1565340243 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:44:08.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.220+0000 7f43c2ffd700 1 -- 172.21.15.59:0/1565340243 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c4102f90 con 0x7f43c4102500 2024-01-23T08:44:08.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.220+0000 7f43c2ffd700 1 --2- 172.21.15.59:0/1565340243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43c4102500 0x7f43c4102910 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f43b40098e0 tx=0x7f43b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7c1a3f814e50d8c server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:08.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.220+0000 7f43c1ffb700 1 -- 172.21.15.59:0/1565340243 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43b400b860 con 0x7f43c4102500 2024-01-23T08:44:08.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.220+0000 7f43c1ffb700 1 -- 172.21.15.59:0/1565340243 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f43b400be70 con 0x7f43c4102500 2024-01-23T08:44:08.391 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.220+0000 7f43c1ffb700 1 -- 172.21.15.59:0/1565340243 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43b4010380 con 0x7f43c4102500 2024-01-23T08:44:08.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.221+0000 7f43c9381700 1 -- 172.21.15.59:0/1565340243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43c4102500 msgr2=0x7f43c4102910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:08.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.221+0000 7f43c9381700 1 --2- 172.21.15.59:0/1565340243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43c4102500 0x7f43c4102910 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f43b40098e0 tx=0x7f43b4009250 comp rx=0 tx=0).stop 2024-01-23T08:44:08.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.221+0000 7f43c9381700 1 -- 172.21.15.59:0/1565340243 shutdown_connections 2024-01-23T08:44:08.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.221+0000 7f43c9381700 1 --2- 172.21.15.59:0/1565340243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43c4102500 0x7f43c4102910 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:08.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.221+0000 7f43c9381700 1 -- 172.21.15.59:0/1565340243 >> 172.21.15.59:0/1565340243 conn(0x7f43c40fdca0 msgr2=0x7f43c41000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:08.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.221+0000 7f43c9381700 1 -- 172.21.15.59:0/1565340243 shutdown_connections 2024-01-23T08:44:08.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.221+0000 7f43c9381700 1 -- 172.21.15.59:0/1565340243 wait complete. 2024-01-23T08:44:08.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.222+0000 7f43c9381700 1 Processor -- start 2024-01-23T08:44:08.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.222+0000 7f43c9381700 1 -- start start 2024-01-23T08:44:08.392 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.222+0000 7f43c9381700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43c4102500 0x7f43c41114b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:08.393 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.222+0000 7f43c9381700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c41119f0 con 0x7f43c4102500 2024-01-23T08:44:08.393 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.222+0000 7f43c2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43c4102500 0x7f43c41114b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:08.393 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.222+0000 7f43c2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43c4102500 0x7f43c41114b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33006/0 (socket says 172.21.15.59:33006) 2024-01-23T08:44:08.393 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.222+0000 7f43c2ffd700 1 -- 172.21.15.59:0/693179325 learned_addr learned my addr 172.21.15.59:0/693179325 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:44:08.393 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.222+0000 7f43c2ffd700 1 -- 172.21.15.59:0/693179325 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b40094f0 con 0x7f43c4102500 2024-01-23T08:44:08.393 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.222+0000 7f43c2ffd700 1 --2- 172.21.15.59:0/693179325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43c4102500 0x7f43c41114b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f43b400bac0 tx=0x7f43b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:08.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.223+0000 7f43ab7fe700 1 -- 172.21.15.59:0/693179325 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43b40109b0 con 0x7f43c4102500 2024-01-23T08:44:08.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.223+0000 7f43ab7fe700 1 -- 172.21.15.59:0/693179325 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f43b401a430 con 0x7f43c4102500 2024-01-23T08:44:08.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.223+0000 7f43ab7fe700 1 -- 172.21.15.59:0/693179325 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43b40245b0 con 0x7f43c4102500 2024-01-23T08:44:08.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.223+0000 7f43c9381700 1 -- 172.21.15.59:0/693179325 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43c4111bf0 con 0x7f43c4102500 2024-01-23T08:44:08.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.223+0000 7f43c9381700 1 -- 172.21.15.59:0/693179325 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43c4112000 con 0x7f43c4102500 2024-01-23T08:44:08.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.223+0000 7f43ab7fe700 1 -- 172.21.15.59:0/693179325 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 12) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f43b4024820 con 0x7f43c4102500 2024-01-23T08:44:08.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.224+0000 7f43ab7fe700 1 --2- 172.21.15.59:0/693179325 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f43ac036640 0x7f43ac038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:08.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.224+0000 7f43ab7fe700 1 -- 172.21.15.59:0/693179325 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f43b404a6a0 con 0x7f43c4102500 2024-01-23T08:44:08.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.224+0000 7f43c27fc700 1 --2- 172.21.15.59:0/693179325 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f43ac036640 0x7f43ac038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:08.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.224+0000 7f43c9381700 1 -- 172.21.15.59:0/693179325 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43c410b3c0 con 0x7f43c4102500 2024-01-23T08:44:08.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.224+0000 7f43c27fc700 1 --2- 172.21.15.59:0/693179325 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f43ac036640 0x7f43ac038af0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f43b8009990 tx=0x7f43b8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:08.394 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.229+0000 7f43ab7fe700 1 -- 172.21.15.59:0/693179325 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f43b4018940 con 0x7f43c4102500 2024-01-23T08:44:08.395 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.443+0000 7f43c9381700 1 -- 172.21.15.59:0/693179325 --> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f43c405f4f0 con 0x7f43ac036640 2024-01-23T08:44:08.395 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.690+0000 7f43ab7fe700 1 -- 172.21.15.59:0/693179325 <== mgr.14162 v2:172.21.15.59:6800/3614124404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f43c405f4f0 con 0x7f43ac036640 2024-01-23T08:44:08.395 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.692+0000 7f43c9381700 1 -- 172.21.15.59:0/693179325 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f43ac036640 msgr2=0x7f43ac038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:08.395 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.692+0000 7f43c9381700 1 --2- 172.21.15.59:0/693179325 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f43ac036640 0x7f43ac038af0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f43b8009990 tx=0x7f43b8006e40 comp rx=0 tx=0).stop 2024-01-23T08:44:08.395 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.692+0000 7f43c9381700 1 -- 172.21.15.59:0/693179325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43c4102500 msgr2=0x7f43c41114b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:08.395 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.692+0000 7f43c9381700 1 --2- 172.21.15.59:0/693179325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43c4102500 0x7f43c41114b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f43b400bac0 tx=0x7f43b4009250 comp rx=0 tx=0).stop 2024-01-23T08:44:08.395 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.692+0000 7f43c9381700 1 -- 172.21.15.59:0/693179325 shutdown_connections 2024-01-23T08:44:08.395 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.692+0000 7f43c9381700 1 --2- 172.21.15.59:0/693179325 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f43ac036640 0x7f43ac038af0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:08.395 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.692+0000 7f43c9381700 1 --2- 172.21.15.59:0/693179325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43c4102500 0x7f43c41114b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:08.395 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.692+0000 7f43c9381700 1 -- 172.21.15.59:0/693179325 >> 172.21.15.59:0/693179325 conn(0x7f43c40fdca0 msgr2=0x7f43c40feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:08.395 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.692+0000 7f43c9381700 1 -- 172.21.15.59:0/693179325 shutdown_connections 2024-01-23T08:44:08.395 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:07.692+0000 7f43c9381700 1 -- 172.21.15.59:0/693179325 wait complete. 2024-01-23T08:44:08.395 INFO:teuthology.orchestra.run.smithi059.stdout:Fetching dashboard port number... 2024-01-23T08:44:09.235 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:08 smithi059 ceph-mon[28424]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:44:09.917 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 8443 2024-01-23T08:44:09.917 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.823+0000 7f537926e700 1 Processor -- start 2024-01-23T08:44:09.917 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.824+0000 7f537926e700 1 -- start start 2024-01-23T08:44:09.917 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.824+0000 7f537926e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5374103070 0x7f5374105450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:09.917 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.824+0000 7f537926e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5374076d70 con 0x7f5374103070 2024-01-23T08:44:09.917 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.824+0000 7f5372ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5374103070 0x7f5374105450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:09.917 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.824+0000 7f5372ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5374103070 0x7f5374105450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55906/0 (socket says 172.21.15.59:55906) 2024-01-23T08:44:09.917 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.824+0000 7f5372ffd700 1 -- 172.21.15.59:0/114712884 learned_addr learned my addr 172.21.15.59:0/114712884 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:44:09.917 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.824+0000 7f5372ffd700 1 -- 172.21.15.59:0/114712884 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5374076eb0 con 0x7f5374103070 2024-01-23T08:44:09.917 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.825+0000 7f5372ffd700 1 --2- 172.21.15.59:0/114712884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5374103070 0x7f5374105450 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f53640098e0 tx=0x7f5364009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a04240923a3411c7 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:09.917 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.825+0000 7f5371ffb700 1 -- 172.21.15.59:0/114712884 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f536400b860 con 0x7f5374103070 2024-01-23T08:44:09.917 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.825+0000 7f5371ffb700 1 -- 172.21.15.59:0/114712884 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f536400be70 con 0x7f5374103070 2024-01-23T08:44:09.917 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.825+0000 7f5371ffb700 1 -- 172.21.15.59:0/114712884 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5364010380 con 0x7f5374103070 2024-01-23T08:44:09.918 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.825+0000 7f537926e700 1 -- 172.21.15.59:0/114712884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5374103070 msgr2=0x7f5374105450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:09.918 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.825+0000 7f537926e700 1 --2- 172.21.15.59:0/114712884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5374103070 0x7f5374105450 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f53640098e0 tx=0x7f5364009250 comp rx=0 tx=0).stop 2024-01-23T08:44:09.918 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.825+0000 7f537926e700 1 -- 172.21.15.59:0/114712884 shutdown_connections 2024-01-23T08:44:09.918 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.825+0000 7f537926e700 1 --2- 172.21.15.59:0/114712884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5374103070 0x7f5374105450 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:09.918 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.825+0000 7f537926e700 1 -- 172.21.15.59:0/114712884 >> 172.21.15.59:0/114712884 conn(0x7f53740fcad0 msgr2=0x7f53740fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:09.918 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.825+0000 7f537926e700 1 -- 172.21.15.59:0/114712884 shutdown_connections 2024-01-23T08:44:09.918 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.825+0000 7f537926e700 1 -- 172.21.15.59:0/114712884 wait complete. 2024-01-23T08:44:09.918 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.826+0000 7f537926e700 1 Processor -- start 2024-01-23T08:44:09.918 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.826+0000 7f537926e700 1 -- start start 2024-01-23T08:44:09.918 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.827+0000 7f537926e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5374103070 0x7f5374102920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:09.918 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.827+0000 7f537926e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5374102e60 con 0x7f5374103070 2024-01-23T08:44:09.918 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.827+0000 7f5372ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5374103070 0x7f5374102920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:09.918 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.827+0000 7f5372ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5374103070 0x7f5374102920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55912/0 (socket says 172.21.15.59:55912) 2024-01-23T08:44:09.918 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.827+0000 7f5372ffd700 1 -- 172.21.15.59:0/1925680503 learned_addr learned my addr 172.21.15.59:0/1925680503 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:44:09.918 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.827+0000 7f5372ffd700 1 -- 172.21.15.59:0/1925680503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53640094f0 con 0x7f5374103070 2024-01-23T08:44:09.918 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.827+0000 7f5372ffd700 1 --2- 172.21.15.59:0/1925680503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5374103070 0x7f5374102920 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f536400c010 tx=0x7f5364009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:09.919 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.827+0000 7f535b7fe700 1 -- 172.21.15.59:0/1925680503 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5364010960 con 0x7f5374103070 2024-01-23T08:44:09.919 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.827+0000 7f535b7fe700 1 -- 172.21.15.59:0/1925680503 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f536401a430 con 0x7f5374103070 2024-01-23T08:44:09.919 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.827+0000 7f537926e700 1 -- 172.21.15.59:0/1925680503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5374100f80 con 0x7f5374103070 2024-01-23T08:44:09.919 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.827+0000 7f535b7fe700 1 -- 172.21.15.59:0/1925680503 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5364019580 con 0x7f5374103070 2024-01-23T08:44:09.919 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.828+0000 7f537926e700 1 -- 172.21.15.59:0/1925680503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53741013f0 con 0x7f5374103070 2024-01-23T08:44:09.919 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.828+0000 7f535b7fe700 1 -- 172.21.15.59:0/1925680503 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 12) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f5364019800 con 0x7f5374103070 2024-01-23T08:44:09.919 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.828+0000 7f537926e700 1 -- 172.21.15.59:0/1925680503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f537405f450 con 0x7f5374103070 2024-01-23T08:44:09.919 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.829+0000 7f535b7fe700 1 --2- 172.21.15.59:0/1925680503 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f535c036640 0x7f535c038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:09.919 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.829+0000 7f535b7fe700 1 -- 172.21.15.59:0/1925680503 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f536404a4a0 con 0x7f5374103070 2024-01-23T08:44:09.919 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.829+0000 7f53727fc700 1 --2- 172.21.15.59:0/1925680503 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f535c036640 0x7f535c038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:09.919 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.829+0000 7f53727fc700 1 --2- 172.21.15.59:0/1925680503 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f535c036640 0x7f535c038af0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5368009990 tx=0x7f5368006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:09.919 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:08.833+0000 7f535b7fe700 1 -- 172.21.15.59:0/1925680503 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f536402bb30 con 0x7f5374103070 2024-01-23T08:44:09.919 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:09.046+0000 7f537926e700 1 -- 172.21.15.59:0/1925680503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f5374106590 con 0x7f5374103070 2024-01-23T08:44:09.919 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:09.046+0000 7f535b7fe700 1 -- 172.21.15.59:0/1925680503 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f5364029e60 con 0x7f5374103070 2024-01-23T08:44:09.920 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:09.048+0000 7f537926e700 1 -- 172.21.15.59:0/1925680503 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f535c036640 msgr2=0x7f535c038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:09.920 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:09.048+0000 7f537926e700 1 --2- 172.21.15.59:0/1925680503 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f535c036640 0x7f535c038af0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5368009990 tx=0x7f5368006e40 comp rx=0 tx=0).stop 2024-01-23T08:44:09.920 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:09.048+0000 7f537926e700 1 -- 172.21.15.59:0/1925680503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5374103070 msgr2=0x7f5374102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:09.920 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:09.048+0000 7f537926e700 1 --2- 172.21.15.59:0/1925680503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5374103070 0x7f5374102920 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f536400c010 tx=0x7f5364009250 comp rx=0 tx=0).stop 2024-01-23T08:44:09.920 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:09.048+0000 7f537926e700 1 -- 172.21.15.59:0/1925680503 shutdown_connections 2024-01-23T08:44:09.920 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:09.048+0000 7f537926e700 1 --2- 172.21.15.59:0/1925680503 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f535c036640 0x7f535c038af0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:09.920 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:09.048+0000 7f537926e700 1 --2- 172.21.15.59:0/1925680503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5374103070 0x7f5374102920 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:09.920 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:09.048+0000 7f537926e700 1 -- 172.21.15.59:0/1925680503 >> 172.21.15.59:0/1925680503 conn(0x7f53740fcad0 msgr2=0x7f53740fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:09.920 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:09.049+0000 7f537926e700 1 -- 172.21.15.59:0/1925680503 shutdown_connections 2024-01-23T08:44:09.920 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:09.049+0000 7f537926e700 1 -- 172.21.15.59:0/1925680503 wait complete. 2024-01-23T08:44:09.927 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-01-23T08:44:09.927 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout disabled 2024-01-23T08:44:09.935 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-01-23T08:44:09.935 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout inactive 2024-01-23T08:44:09.935 INFO:teuthology.orchestra.run.smithi059.stdout:firewalld.service is not enabled 2024-01-23T08:44:09.935 INFO:teuthology.orchestra.run.smithi059.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-01-23T08:44:09.937 INFO:teuthology.orchestra.run.smithi059.stdout:Ceph Dashboard is now available at: 2024-01-23T08:44:09.937 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:44:09.937 INFO:teuthology.orchestra.run.smithi059.stdout: URL: https://smithi059.front.sepia.ceph.com:8443/ 2024-01-23T08:44:09.937 INFO:teuthology.orchestra.run.smithi059.stdout: User: admin 2024-01-23T08:44:09.938 INFO:teuthology.orchestra.run.smithi059.stdout: Password: 4bc31e2rti 2024-01-23T08:44:09.938 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:44:09.938 INFO:teuthology.orchestra.run.smithi059.stdout:Saving cluster configuration to /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/config directory 2024-01-23T08:44:09.938 INFO:teuthology.orchestra.run.smithi059.stdout:Enabling autotune for osd_memory_target 2024-01-23T08:44:10.174 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:09 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/1925680503' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-01-23T08:44:11.182 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:10 smithi059 ceph-mon[28424]: mgrmap e13: smithi059.bzqirf(active, since 6s) 2024-01-23T08:44:11.189 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.383+0000 7f646b9e0700 1 Processor -- start 2024-01-23T08:44:11.189 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.383+0000 7f646b9e0700 1 -- start start 2024-01-23T08:44:11.189 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.383+0000 7f646b9e0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6464076c80 0x7f6464105090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:11.189 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.383+0000 7f646b9e0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64641055d0 con 0x7f6464076c80 2024-01-23T08:44:11.189 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.383+0000 7f646977c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6464076c80 0x7f6464105090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:11.189 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.383+0000 7f646977c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6464076c80 0x7f6464105090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55924/0 (socket says 172.21.15.59:55924) 2024-01-23T08:44:11.189 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.383+0000 7f646977c700 1 -- 172.21.15.59:0/193237363 learned_addr learned my addr 172.21.15.59:0/193237363 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:44:11.189 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.383+0000 7f646977c700 1 -- 172.21.15.59:0/193237363 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6464105710 con 0x7f6464076c80 2024-01-23T08:44:11.189 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.384+0000 7f646977c700 1 --2- 172.21.15.59:0/193237363 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6464076c80 0x7f6464105090 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f64540098e0 tx=0x7f6454009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dc48175090ea7c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:11.189 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.384+0000 7f645bfff700 1 -- 172.21.15.59:0/193237363 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f645400b860 con 0x7f6464076c80 2024-01-23T08:44:11.189 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.384+0000 7f645bfff700 1 -- 172.21.15.59:0/193237363 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f645400be70 con 0x7f6464076c80 2024-01-23T08:44:11.190 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.384+0000 7f645bfff700 1 -- 172.21.15.59:0/193237363 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6454010380 con 0x7f6464076c80 2024-01-23T08:44:11.190 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.384+0000 7f646b9e0700 1 -- 172.21.15.59:0/193237363 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6464076c80 msgr2=0x7f6464105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:11.190 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.384+0000 7f646b9e0700 1 --2- 172.21.15.59:0/193237363 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6464076c80 0x7f6464105090 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f64540098e0 tx=0x7f6454009250 comp rx=0 tx=0).stop 2024-01-23T08:44:11.190 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.384+0000 7f646b9e0700 1 -- 172.21.15.59:0/193237363 shutdown_connections 2024-01-23T08:44:11.190 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.384+0000 7f646b9e0700 1 --2- 172.21.15.59:0/193237363 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6464076c80 0x7f6464105090 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:11.190 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.384+0000 7f646b9e0700 1 -- 172.21.15.59:0/193237363 >> 172.21.15.59:0/193237363 conn(0x7f64640fcb40 msgr2=0x7f64640fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:11.190 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.385+0000 7f646b9e0700 1 -- 172.21.15.59:0/193237363 shutdown_connections 2024-01-23T08:44:11.190 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.385+0000 7f646b9e0700 1 -- 172.21.15.59:0/193237363 wait complete. 2024-01-23T08:44:11.190 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.385+0000 7f646b9e0700 1 Processor -- start 2024-01-23T08:44:11.190 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.385+0000 7f646b9e0700 1 -- start start 2024-01-23T08:44:11.190 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.385+0000 7f646b9e0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6464076c80 0x7f6464102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:11.190 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.385+0000 7f646b9e0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6464102e70 con 0x7f6464076c80 2024-01-23T08:44:11.190 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.385+0000 7f646977c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6464076c80 0x7f6464102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:11.190 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.385+0000 7f646977c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6464076c80 0x7f6464102930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55938/0 (socket says 172.21.15.59:55938) 2024-01-23T08:44:11.190 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.385+0000 7f646977c700 1 -- 172.21.15.59:0/373937627 learned_addr learned my addr 172.21.15.59:0/373937627 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:44:11.190 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.386+0000 7f646977c700 1 -- 172.21.15.59:0/373937627 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64540094f0 con 0x7f6464076c80 2024-01-23T08:44:11.191 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.386+0000 7f646977c700 1 --2- 172.21.15.59:0/373937627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6464076c80 0x7f6464102930 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f645400ba70 tx=0x7f6454009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:11.191 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.386+0000 7f6459ffb700 1 -- 172.21.15.59:0/373937627 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6454010960 con 0x7f6464076c80 2024-01-23T08:44:11.191 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.386+0000 7f6459ffb700 1 -- 172.21.15.59:0/373937627 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f645401a430 con 0x7f6464076c80 2024-01-23T08:44:11.191 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.386+0000 7f646b9e0700 1 -- 172.21.15.59:0/373937627 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6464100f90 con 0x7f6464076c80 2024-01-23T08:44:11.191 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.386+0000 7f6459ffb700 1 -- 172.21.15.59:0/373937627 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64540245b0 con 0x7f6464076c80 2024-01-23T08:44:11.191 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.386+0000 7f646b9e0700 1 -- 172.21.15.59:0/373937627 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6464101400 con 0x7f6464076c80 2024-01-23T08:44:11.191 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.387+0000 7f6459ffb700 1 -- 172.21.15.59:0/373937627 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f6454010ac0 con 0x7f6464076c80 2024-01-23T08:44:11.191 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.387+0000 7f6459ffb700 1 --2- 172.21.15.59:0/373937627 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f64500366e0 0x7f6450038b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:11.191 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.387+0000 7f646b9e0700 1 -- 172.21.15.59:0/373937627 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64641064f0 con 0x7f6464076c80 2024-01-23T08:44:11.191 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.387+0000 7f6459ffb700 1 -- 172.21.15.59:0/373937627 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f645404a360 con 0x7f6464076c80 2024-01-23T08:44:11.191 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.387+0000 7f6468f7b700 1 --2- 172.21.15.59:0/373937627 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f64500366e0 0x7f6450038b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:11.191 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.388+0000 7f6468f7b700 1 --2- 172.21.15.59:0/373937627 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f64500366e0 0x7f6450038b90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6460009990 tx=0x7f6460006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:11.191 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.392+0000 7f6459ffb700 1 -- 172.21.15.59:0/373937627 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f645402b8e0 con 0x7f6464076c80 2024-01-23T08:44:11.191 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.584+0000 7f646b9e0700 1 -- 172.21.15.59:0/373937627 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f6464101b30 con 0x7f6464076c80 2024-01-23T08:44:11.191 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.584+0000 7f6459ffb700 1 -- 172.21.15.59:0/373937627 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f64540166e0 con 0x7f6464076c80 2024-01-23T08:44:11.191 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.587+0000 7f646b9e0700 1 -- 172.21.15.59:0/373937627 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f64500366e0 msgr2=0x7f6450038b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:11.192 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.587+0000 7f646b9e0700 1 --2- 172.21.15.59:0/373937627 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f64500366e0 0x7f6450038b90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6460009990 tx=0x7f6460006e40 comp rx=0 tx=0).stop 2024-01-23T08:44:11.192 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.587+0000 7f646b9e0700 1 -- 172.21.15.59:0/373937627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6464076c80 msgr2=0x7f6464102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:11.192 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.587+0000 7f646b9e0700 1 --2- 172.21.15.59:0/373937627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6464076c80 0x7f6464102930 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f645400ba70 tx=0x7f6454009250 comp rx=0 tx=0).stop 2024-01-23T08:44:11.192 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.587+0000 7f646b9e0700 1 -- 172.21.15.59:0/373937627 shutdown_connections 2024-01-23T08:44:11.192 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.587+0000 7f646b9e0700 1 --2- 172.21.15.59:0/373937627 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f64500366e0 0x7f6450038b90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:11.192 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.587+0000 7f646b9e0700 1 --2- 172.21.15.59:0/373937627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6464076c80 0x7f6464102930 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:11.192 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.587+0000 7f646b9e0700 1 -- 172.21.15.59:0/373937627 >> 172.21.15.59:0/373937627 conn(0x7f64640fcb40 msgr2=0x7f64640fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:11.192 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.587+0000 7f646b9e0700 1 -- 172.21.15.59:0/373937627 shutdown_connections 2024-01-23T08:44:11.192 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:10.587+0000 7f646b9e0700 1 -- 172.21.15.59:0/373937627 wait complete. 2024-01-23T08:44:12.462 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.565+0000 7f5404505700 1 Processor -- start 2024-01-23T08:44:12.462 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.565+0000 7f5404505700 1 -- start start 2024-01-23T08:44:12.462 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.565+0000 7f5404505700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53fc102500 0x7f53fc102910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:12.462 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.565+0000 7f5404505700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53fc102e50 con 0x7f53fc102500 2024-01-23T08:44:12.462 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.566+0000 7f54022a1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53fc102500 0x7f53fc102910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:12.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.566+0000 7f54022a1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53fc102500 0x7f53fc102910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55942/0 (socket says 172.21.15.59:55942) 2024-01-23T08:44:12.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.566+0000 7f54022a1700 1 -- 172.21.15.59:0/2030748098 learned_addr learned my addr 172.21.15.59:0/2030748098 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:44:12.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.566+0000 7f54022a1700 1 -- 172.21.15.59:0/2030748098 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53fc102f90 con 0x7f53fc102500 2024-01-23T08:44:12.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.566+0000 7f54022a1700 1 --2- 172.21.15.59:0/2030748098 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53fc102500 0x7f53fc102910 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f53ec009e50 tx=0x7f53ec009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fa70ac760ceff5a server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:12.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.566+0000 7f540129f700 1 -- 172.21.15.59:0/2030748098 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53ec0092e0 con 0x7f53fc102500 2024-01-23T08:44:12.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.566+0000 7f540129f700 1 -- 172.21.15.59:0/2030748098 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f53ec012430 con 0x7f53fc102500 2024-01-23T08:44:12.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.566+0000 7f540129f700 1 -- 172.21.15.59:0/2030748098 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53ec0075c0 con 0x7f53fc102500 2024-01-23T08:44:12.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.567+0000 7f5404505700 1 -- 172.21.15.59:0/2030748098 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53fc102500 msgr2=0x7f53fc102910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:12.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.567+0000 7f5404505700 1 --2- 172.21.15.59:0/2030748098 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53fc102500 0x7f53fc102910 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f53ec009e50 tx=0x7f53ec009840 comp rx=0 tx=0).stop 2024-01-23T08:44:12.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.567+0000 7f5404505700 1 -- 172.21.15.59:0/2030748098 shutdown_connections 2024-01-23T08:44:12.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.567+0000 7f5404505700 1 --2- 172.21.15.59:0/2030748098 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53fc102500 0x7f53fc102910 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:12.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.567+0000 7f5404505700 1 -- 172.21.15.59:0/2030748098 >> 172.21.15.59:0/2030748098 conn(0x7f53fc0fdc80 msgr2=0x7f53fc100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:12.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.567+0000 7f5404505700 1 -- 172.21.15.59:0/2030748098 shutdown_connections 2024-01-23T08:44:12.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.567+0000 7f5404505700 1 -- 172.21.15.59:0/2030748098 wait complete. 2024-01-23T08:44:12.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.567+0000 7f5404505700 1 Processor -- start 2024-01-23T08:44:12.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.568+0000 7f5404505700 1 -- start start 2024-01-23T08:44:12.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.568+0000 7f5404505700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53fc102500 0x7f53fc078b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:12.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.568+0000 7f5404505700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53fc079090 con 0x7f53fc102500 2024-01-23T08:44:12.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.568+0000 7f54022a1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53fc102500 0x7f53fc078b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:12.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.568+0000 7f54022a1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53fc102500 0x7f53fc078b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55944/0 (socket says 172.21.15.59:55944) 2024-01-23T08:44:12.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.568+0000 7f54022a1700 1 -- 172.21.15.59:0/1330204942 learned_addr learned my addr 172.21.15.59:0/1330204942 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:44:12.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.568+0000 7f54022a1700 1 -- 172.21.15.59:0/1330204942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ec009ae0 con 0x7f53fc102500 2024-01-23T08:44:12.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.568+0000 7f54022a1700 1 --2- 172.21.15.59:0/1330204942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53fc102500 0x7f53fc078b50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f53ec0127d0 tx=0x7f53ec009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:12.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.568+0000 7f53faffd700 1 -- 172.21.15.59:0/1330204942 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53ec0092e0 con 0x7f53fc102500 2024-01-23T08:44:12.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.568+0000 7f53faffd700 1 -- 172.21.15.59:0/1330204942 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f53ec007ce0 con 0x7f53fc102500 2024-01-23T08:44:12.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.568+0000 7f5404505700 1 -- 172.21.15.59:0/1330204942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53fc0771b0 con 0x7f53fc102500 2024-01-23T08:44:12.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.568+0000 7f53faffd700 1 -- 172.21.15.59:0/1330204942 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53ec01c910 con 0x7f53fc102500 2024-01-23T08:44:12.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.568+0000 7f5404505700 1 -- 172.21.15.59:0/1330204942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53fc077620 con 0x7f53fc102500 2024-01-23T08:44:12.464 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.569+0000 7f53faffd700 1 -- 172.21.15.59:0/1330204942 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f53ec019070 con 0x7f53fc102500 2024-01-23T08:44:12.465 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.569+0000 7f53faffd700 1 --2- 172.21.15.59:0/1330204942 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f53e40366e0 0x7f53e4038b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:12.465 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.570+0000 7f53faffd700 1 -- 172.21.15.59:0/1330204942 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f53ec049e90 con 0x7f53fc102500 2024-01-23T08:44:12.465 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.570+0000 7f5404505700 1 -- 172.21.15.59:0/1330204942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53fc05f450 con 0x7f53fc102500 2024-01-23T08:44:12.465 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.570+0000 7f5401aa0700 1 --2- 172.21.15.59:0/1330204942 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f53e40366e0 0x7f53e4038b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:12.465 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.570+0000 7f5401aa0700 1 --2- 172.21.15.59:0/1330204942 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f53e40366e0 0x7f53e4038b90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f53f0009990 tx=0x7f53f0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:12.465 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.574+0000 7f53faffd700 1 -- 172.21.15.59:0/1330204942 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f53ec017080 con 0x7f53fc102500 2024-01-23T08:44:12.465 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.876+0000 7f5404505700 1 -- 172.21.15.59:0/1330204942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f53fc064940 con 0x7f53fc102500 2024-01-23T08:44:12.465 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.878+0000 7f53faffd700 1 -- 172.21.15.59:0/1330204942 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f53ec02a030 con 0x7f53fc102500 2024-01-23T08:44:12.465 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.880+0000 7f5404505700 1 -- 172.21.15.59:0/1330204942 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f53e40366e0 msgr2=0x7f53e4038b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:12.465 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.880+0000 7f5404505700 1 --2- 172.21.15.59:0/1330204942 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f53e40366e0 0x7f53e4038b90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f53f0009990 tx=0x7f53f0006e40 comp rx=0 tx=0).stop 2024-01-23T08:44:12.465 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.880+0000 7f5404505700 1 -- 172.21.15.59:0/1330204942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53fc102500 msgr2=0x7f53fc078b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:12.465 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.880+0000 7f5404505700 1 --2- 172.21.15.59:0/1330204942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53fc102500 0x7f53fc078b50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f53ec0127d0 tx=0x7f53ec009840 comp rx=0 tx=0).stop 2024-01-23T08:44:12.465 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.880+0000 7f5404505700 1 -- 172.21.15.59:0/1330204942 shutdown_connections 2024-01-23T08:44:12.466 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.880+0000 7f5404505700 1 --2- 172.21.15.59:0/1330204942 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f53e40366e0 0x7f53e4038b90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:12.466 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.880+0000 7f5404505700 1 --2- 172.21.15.59:0/1330204942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53fc102500 0x7f53fc078b50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:12.466 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.880+0000 7f5404505700 1 -- 172.21.15.59:0/1330204942 >> 172.21.15.59:0/1330204942 conn(0x7f53fc0fdc80 msgr2=0x7f53fc0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:12.466 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.880+0000 7f5404505700 1 -- 172.21.15.59:0/1330204942 shutdown_connections 2024-01-23T08:44:12.466 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2024-01-23T08:44:11.880+0000 7f5404505700 1 -- 172.21.15.59:0/1330204942 wait complete. 2024-01-23T08:44:12.466 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-01-23T08:44:12.466 INFO:teuthology.orchestra.run.smithi059.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-01-23T08:44:12.466 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:44:12.466 INFO:teuthology.orchestra.run.smithi059.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-01-23T08:44:12.466 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:44:12.466 INFO:teuthology.orchestra.run.smithi059.stdout:Or, if you are only running a single cluster on this host: 2024-01-23T08:44:12.466 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:44:12.466 INFO:teuthology.orchestra.run.smithi059.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-01-23T08:44:12.466 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:44:12.466 INFO:teuthology.orchestra.run.smithi059.stdout:Please consider enabling telemetry to help improve Ceph: 2024-01-23T08:44:12.466 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:44:12.466 INFO:teuthology.orchestra.run.smithi059.stdout: ceph telemetry on 2024-01-23T08:44:12.467 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:44:12.467 INFO:teuthology.orchestra.run.smithi059.stdout:For more information see: 2024-01-23T08:44:12.467 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:44:12.467 INFO:teuthology.orchestra.run.smithi059.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-01-23T08:44:12.467 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:44:12.467 INFO:teuthology.orchestra.run.smithi059.stdout:Bootstrap complete. 2024-01-23T08:44:12.504 INFO:tasks.cephadm:Fetching config... 2024-01-23T08:44:12.505 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-23T08:44:12.505 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-01-23T08:44:12.563 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-01-23T08:44:12.564 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-23T08:44:12.564 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-01-23T08:44:12.618 INFO:tasks.cephadm:Fetching mon keyring... 2024-01-23T08:44:12.618 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-23T08:44:12.618 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/keyring of=/dev/stdout 2024-01-23T08:44:12.684 INFO:tasks.cephadm:Fetching pub ssh key... 2024-01-23T08:44:12.685 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-23T08:44:12.685 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-01-23T08:44:12.740 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-01-23T08:44:12.740 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCZRpFiSdBf0oMP09S9Y1pzIjzVFOtgF6EM2wE+dTtB6bGYhBlz9lrEvoVQdFAdhlPYVoV2uiIQGkmXAuPvp+sAqqjjZ+842+L0TKRakwwYXZaKVkiJ174NUWhTlAviPc2BAAeTRZTrsXi8Duwd/jdfegi/Nu9GQl2FO13ARWXUUHJPxX8zUe81Tv0rOGjZyQe/2iTsXrdWpHeBIk0PLvnw3Vaq0ttKruOQkRs4ikc1uOl1e3iIeVrnC/DEDS/xl2Vm6adh2phaMh2Wo9ngmkNzLLjNbiXU4BePS8G8RrF3cgdBojbW8fRovi2RLgEA1WKZ6hOthH3b90AkKnyx9k6DcS6Y3jsdPMxL3x3gJV9fXRlCPLPy+CCrC1aY2udIHE6y1DSMRCpdg3oAj2C3M4Vy7CVZ5fMIygt2hqYwpw1MMCUui5kQbW+DeTuy3hdSDA9MSB1Ok+FbW1kCLlJmIanzxtG33J651IJuDwHswF0PFilNy++3tzntKCL33rJ/wnc= ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-23T08:44:12.826 INFO:teuthology.orchestra.run.smithi059.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCZRpFiSdBf0oMP09S9Y1pzIjzVFOtgF6EM2wE+dTtB6bGYhBlz9lrEvoVQdFAdhlPYVoV2uiIQGkmXAuPvp+sAqqjjZ+842+L0TKRakwwYXZaKVkiJ174NUWhTlAviPc2BAAeTRZTrsXi8Duwd/jdfegi/Nu9GQl2FO13ARWXUUHJPxX8zUe81Tv0rOGjZyQe/2iTsXrdWpHeBIk0PLvnw3Vaq0ttKruOQkRs4ikc1uOl1e3iIeVrnC/DEDS/xl2Vm6adh2phaMh2Wo9ngmkNzLLjNbiXU4BePS8G8RrF3cgdBojbW8fRovi2RLgEA1WKZ6hOthH3b90AkKnyx9k6DcS6Y3jsdPMxL3x3gJV9fXRlCPLPy+CCrC1aY2udIHE6y1DSMRCpdg3oAj2C3M4Vy7CVZ5fMIygt2hqYwpw1MMCUui5kQbW+DeTuy3hdSDA9MSB1Ok+FbW1kCLlJmIanzxtG33J651IJuDwHswF0PFilNy++3tzntKCL33rJ/wnc= ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715 2024-01-23T08:44:12.842 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCZRpFiSdBf0oMP09S9Y1pzIjzVFOtgF6EM2wE+dTtB6bGYhBlz9lrEvoVQdFAdhlPYVoV2uiIQGkmXAuPvp+sAqqjjZ+842+L0TKRakwwYXZaKVkiJ174NUWhTlAviPc2BAAeTRZTrsXi8Duwd/jdfegi/Nu9GQl2FO13ARWXUUHJPxX8zUe81Tv0rOGjZyQe/2iTsXrdWpHeBIk0PLvnw3Vaq0ttKruOQkRs4ikc1uOl1e3iIeVrnC/DEDS/xl2Vm6adh2phaMh2Wo9ngmkNzLLjNbiXU4BePS8G8RrF3cgdBojbW8fRovi2RLgEA1WKZ6hOthH3b90AkKnyx9k6DcS6Y3jsdPMxL3x3gJV9fXRlCPLPy+CCrC1aY2udIHE6y1DSMRCpdg3oAj2C3M4Vy7CVZ5fMIygt2hqYwpw1MMCUui5kQbW+DeTuy3hdSDA9MSB1Ok+FbW1kCLlJmIanzxtG33J651IJuDwHswF0PFilNy++3tzntKCL33rJ/wnc= ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-23T08:44:12.891 INFO:teuthology.orchestra.run.smithi137.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCZRpFiSdBf0oMP09S9Y1pzIjzVFOtgF6EM2wE+dTtB6bGYhBlz9lrEvoVQdFAdhlPYVoV2uiIQGkmXAuPvp+sAqqjjZ+842+L0TKRakwwYXZaKVkiJ174NUWhTlAviPc2BAAeTRZTrsXi8Duwd/jdfegi/Nu9GQl2FO13ARWXUUHJPxX8zUe81Tv0rOGjZyQe/2iTsXrdWpHeBIk0PLvnw3Vaq0ttKruOQkRs4ikc1uOl1e3iIeVrnC/DEDS/xl2Vm6adh2phaMh2Wo9ngmkNzLLjNbiXU4BePS8G8RrF3cgdBojbW8fRovi2RLgEA1WKZ6hOthH3b90AkKnyx9k6DcS6Y3jsdPMxL3x3gJV9fXRlCPLPy+CCrC1aY2udIHE6y1DSMRCpdg3oAj2C3M4Vy7CVZ5fMIygt2hqYwpw1MMCUui5kQbW+DeTuy3hdSDA9MSB1Ok+FbW1kCLlJmIanzxtG33J651IJuDwHswF0PFilNy++3tzntKCL33rJ/wnc= ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715 2024-01-23T08:44:12.906 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-01-23T08:44:13.157 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:44:13.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:12 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/1330204942' entity='client.admin' 2024-01-23T08:44:15.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.715+0000 7f67611b5700 1 -- 172.21.15.59:0/3857689470 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f675c066510 msgr2=0x7f675c0ff600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:15.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.715+0000 7f67611b5700 1 --2- 172.21.15.59:0/3857689470 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f675c066510 0x7f675c0ff600 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f674c009c60 tx=0x7f674c004d10 comp rx=0 tx=0).stop 2024-01-23T08:44:15.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.715+0000 7f67611b5700 1 -- 172.21.15.59:0/3857689470 shutdown_connections 2024-01-23T08:44:15.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.715+0000 7f67611b5700 1 --2- 172.21.15.59:0/3857689470 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f675c066510 0x7f675c0ff600 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:15.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.715+0000 7f67611b5700 1 -- 172.21.15.59:0/3857689470 >> 172.21.15.59:0/3857689470 conn(0x7f675c0f7030 msgr2=0x7f675c0f9440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:15.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.715+0000 7f67611b5700 1 -- 172.21.15.59:0/3857689470 shutdown_connections 2024-01-23T08:44:15.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.715+0000 7f67611b5700 1 -- 172.21.15.59:0/3857689470 wait complete. 2024-01-23T08:44:15.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.715+0000 7f67611b5700 1 Processor -- start 2024-01-23T08:44:15.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.716+0000 7f67611b5700 1 -- start start 2024-01-23T08:44:15.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.716+0000 7f67611b5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f675c066510 0x7f675c0fd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.716+0000 7f67611b5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675c0fb500 con 0x7f675c066510 2024-01-23T08:44:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.716+0000 7f675ad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f675c066510 0x7f675c0fd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.716+0000 7f675ad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f675c066510 0x7f675c0fd080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55970/0 (socket says 172.21.15.59:55970) 2024-01-23T08:44:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.716+0000 7f675ad9d700 1 -- 172.21.15.59:0/3475701678 learned_addr learned my addr 172.21.15.59:0/3475701678 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:44:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.716+0000 7f675ad9d700 1 -- 172.21.15.59:0/3475701678 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674c009940 con 0x7f675c066510 2024-01-23T08:44:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.716+0000 7f675ad9d700 1 --2- 172.21.15.59:0/3475701678 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f675c066510 0x7f675c0fd080 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f674c007b30 tx=0x7f674c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.716+0000 7f67537fe700 1 -- 172.21.15.59:0/3475701678 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f674c00bec0 con 0x7f675c066510 2024-01-23T08:44:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.716+0000 7f67611b5700 1 -- 172.21.15.59:0/3475701678 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f675c0fb700 con 0x7f675c066510 2024-01-23T08:44:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.716+0000 7f67537fe700 1 -- 172.21.15.59:0/3475701678 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f674c01d8f0 con 0x7f675c066510 2024-01-23T08:44:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.716+0000 7f67537fe700 1 -- 172.21.15.59:0/3475701678 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f674c0128e0 con 0x7f675c066510 2024-01-23T08:44:15.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.717+0000 7f67611b5700 1 -- 172.21.15.59:0/3475701678 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f675c0fbbf0 con 0x7f675c066510 2024-01-23T08:44:15.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.717+0000 7f67537fe700 1 -- 172.21.15.59:0/3475701678 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f674c019040 con 0x7f675c066510 2024-01-23T08:44:15.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.717+0000 7f67611b5700 1 -- 172.21.15.59:0/3475701678 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f675c1066d0 con 0x7f675c066510 2024-01-23T08:44:15.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.718+0000 7f67537fe700 1 --2- 172.21.15.59:0/3475701678 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f67480366d0 0x7f6748038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:15.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.718+0000 7f67537fe700 1 -- 172.21.15.59:0/3475701678 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f674c04a190 con 0x7f675c066510 2024-01-23T08:44:15.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.718+0000 7f675a59c700 1 --2- 172.21.15.59:0/3475701678 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f67480366d0 0x7f6748038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:15.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.718+0000 7f675a59c700 1 --2- 172.21.15.59:0/3475701678 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f67480366d0 0x7f6748038b80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6744009990 tx=0x7f6744006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:15.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.722+0000 7f67537fe700 1 -- 172.21.15.59:0/3475701678 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f674c02ebb0 con 0x7f675c066510 2024-01-23T08:44:15.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.912+0000 7f67611b5700 1 -- 172.21.15.59:0/3475701678 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f675c1026c0 con 0x7f675c066510 2024-01-23T08:44:15.910 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:15 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:15.910 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:15 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:15.910 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:15 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd/host:smithi059", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:44:15.910 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:15 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:15.910 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:15 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi059", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-23T08:44:15.911 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:15 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi059", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-23T08:44:15.911 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:15 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:44:15.911 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:15 smithi059 ceph-mon[28424]: Deploying daemon crash.smithi059 on smithi059 2024-01-23T08:44:15.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.916+0000 7f67537fe700 1 -- 172.21.15.59:0/3475701678 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f674c021020 con 0x7f675c066510 2024-01-23T08:44:15.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.919+0000 7f67611b5700 1 -- 172.21.15.59:0/3475701678 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f67480366d0 msgr2=0x7f6748038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:15.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.919+0000 7f67611b5700 1 --2- 172.21.15.59:0/3475701678 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f67480366d0 0x7f6748038b80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6744009990 tx=0x7f6744006e40 comp rx=0 tx=0).stop 2024-01-23T08:44:15.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.919+0000 7f67611b5700 1 -- 172.21.15.59:0/3475701678 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f675c066510 msgr2=0x7f675c0fd080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:15.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.919+0000 7f67611b5700 1 --2- 172.21.15.59:0/3475701678 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f675c066510 0x7f675c0fd080 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f674c007b30 tx=0x7f674c004d10 comp rx=0 tx=0).stop 2024-01-23T08:44:15.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.919+0000 7f67611b5700 1 -- 172.21.15.59:0/3475701678 shutdown_connections 2024-01-23T08:44:15.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.919+0000 7f67611b5700 1 --2- 172.21.15.59:0/3475701678 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f67480366d0 0x7f6748038b80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:15.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.919+0000 7f67611b5700 1 --2- 172.21.15.59:0/3475701678 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f675c066510 0x7f675c0fd080 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:15.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.919+0000 7f67611b5700 1 -- 172.21.15.59:0/3475701678 >> 172.21.15.59:0/3475701678 conn(0x7f675c0f7030 msgr2=0x7f675c0f7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:15.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.919+0000 7f67611b5700 1 -- 172.21.15.59:0/3475701678 shutdown_connections 2024-01-23T08:44:15.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:15.919+0000 7f67611b5700 1 -- 172.21.15.59:0/3475701678 wait complete. 2024-01-23T08:44:16.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:16 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/3475701678' entity='client.admin' 2024-01-23T08:44:17.186 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-01-23T08:44:17.186 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-01-23T08:44:18.341 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:44:19.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.917+0000 7f26820f4700 1 -- 172.21.15.59:0/2251611923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f267c0fca50 msgr2=0x7f267c0fce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:19.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.917+0000 7f26820f4700 1 --2- 172.21.15.59:0/2251611923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f267c0fca50 0x7f267c0fce60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f266c0048c0 tx=0x7f266c009e30 comp rx=0 tx=0).stop 2024-01-23T08:44:19.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.917+0000 7f26820f4700 1 -- 172.21.15.59:0/2251611923 shutdown_connections 2024-01-23T08:44:19.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.917+0000 7f26820f4700 1 --2- 172.21.15.59:0/2251611923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f267c0fca50 0x7f267c0fce60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:19.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.917+0000 7f26820f4700 1 -- 172.21.15.59:0/2251611923 >> 172.21.15.59:0/2251611923 conn(0x7f267c0f8130 msgr2=0x7f267c0fa540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:19.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.917+0000 7f26820f4700 1 -- 172.21.15.59:0/2251611923 shutdown_connections 2024-01-23T08:44:19.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.917+0000 7f26820f4700 1 -- 172.21.15.59:0/2251611923 wait complete. 2024-01-23T08:44:19.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.917+0000 7f26820f4700 1 Processor -- start 2024-01-23T08:44:19.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.918+0000 7f26820f4700 1 -- start start 2024-01-23T08:44:19.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.918+0000 7f26820f4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f267c0fca50 0x7f267c10cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:19.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.918+0000 7f26820f4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267c10d280 con 0x7f267c0fca50 2024-01-23T08:44:19.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.918+0000 7f267b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f267c0fca50 0x7f267c10cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:19.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.918+0000 7f267b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f267c0fca50 0x7f267c10cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36644/0 (socket says 172.21.15.59:36644) 2024-01-23T08:44:19.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.918+0000 7f267b7fe700 1 -- 172.21.15.59:0/2582378119 learned_addr learned my addr 172.21.15.59:0/2582378119 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:44:19.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.918+0000 7f267b7fe700 1 -- 172.21.15.59:0/2582378119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f266c005040 con 0x7f267c0fca50 2024-01-23T08:44:19.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.918+0000 7f267b7fe700 1 --2- 172.21.15.59:0/2582378119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f267c0fca50 0x7f267c10cd40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f266c00b690 tx=0x7f266c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:19.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.919+0000 7f2663fff700 1 -- 172.21.15.59:0/2582378119 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f266c009650 con 0x7f267c0fca50 2024-01-23T08:44:19.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.919+0000 7f26820f4700 1 -- 172.21.15.59:0/2582378119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f267c10d480 con 0x7f267c0fca50 2024-01-23T08:44:19.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.919+0000 7f2663fff700 1 -- 172.21.15.59:0/2582378119 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f266c0097b0 con 0x7f267c0fca50 2024-01-23T08:44:19.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.919+0000 7f2663fff700 1 -- 172.21.15.59:0/2582378119 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f266c01c5d0 con 0x7f267c0fca50 2024-01-23T08:44:19.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.919+0000 7f26820f4700 1 -- 172.21.15.59:0/2582378119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f267c10dfe0 con 0x7f267c0fca50 2024-01-23T08:44:19.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.920+0000 7f2663fff700 1 -- 172.21.15.59:0/2582378119 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f266c01c840 con 0x7f267c0fca50 2024-01-23T08:44:19.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.920+0000 7f2663fff700 1 --2- 172.21.15.59:0/2582378119 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f26640366d0 0x7f2664038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:19.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.920+0000 7f2663fff700 1 -- 172.21.15.59:0/2582378119 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f266c049c80 con 0x7f267c0fca50 2024-01-23T08:44:19.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.920+0000 7f267affd700 1 --2- 172.21.15.59:0/2582378119 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f26640366d0 0x7f2664038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:19.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.920+0000 7f26820f4700 1 -- 172.21.15.59:0/2582378119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f267c04bd60 con 0x7f267c0fca50 2024-01-23T08:44:19.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.920+0000 7f267affd700 1 --2- 172.21.15.59:0/2582378119 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f26640366d0 0x7f2664038b80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2670009990 tx=0x7f2670006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:19.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:19.925+0000 7f2663fff700 1 -- 172.21.15.59:0/2582378119 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f266c023950 con 0x7f267c0fca50 2024-01-23T08:44:20.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:20.124+0000 7f26820f4700 1 -- 172.21.15.59:0/2582378119 --> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f267c10ea80 con 0x7f26640366d0 2024-01-23T08:44:20.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:20.128+0000 7f2663fff700 1 -- 172.21.15.59:0/2582378119 <== mgr.14162 v2:172.21.15.59:6800/3614124404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f267c10ea80 con 0x7f26640366d0 2024-01-23T08:44:20.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:20.129+0000 7f26820f4700 1 -- 172.21.15.59:0/2582378119 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f26640366d0 msgr2=0x7f2664038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:20.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:20.129+0000 7f26820f4700 1 --2- 172.21.15.59:0/2582378119 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f26640366d0 0x7f2664038b80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2670009990 tx=0x7f2670006e40 comp rx=0 tx=0).stop 2024-01-23T08:44:20.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:20.130+0000 7f26820f4700 1 -- 172.21.15.59:0/2582378119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f267c0fca50 msgr2=0x7f267c10cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:20.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:20.130+0000 7f26820f4700 1 --2- 172.21.15.59:0/2582378119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f267c0fca50 0x7f267c10cd40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f266c00b690 tx=0x7f266c009e30 comp rx=0 tx=0).stop 2024-01-23T08:44:20.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:20.130+0000 7f26820f4700 1 -- 172.21.15.59:0/2582378119 shutdown_connections 2024-01-23T08:44:20.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:20.130+0000 7f26820f4700 1 --2- 172.21.15.59:0/2582378119 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f26640366d0 0x7f2664038b80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:20.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:20.130+0000 7f26820f4700 1 --2- 172.21.15.59:0/2582378119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f267c0fca50 0x7f267c10cd40 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:20.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:20.130+0000 7f26820f4700 1 -- 172.21.15.59:0/2582378119 >> 172.21.15.59:0/2582378119 conn(0x7f267c0f8130 msgr2=0x7f267c0f8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:20.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:20.130+0000 7f26820f4700 1 -- 172.21.15.59:0/2582378119 shutdown_connections 2024-01-23T08:44:20.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:20.130+0000 7f26820f4700 1 -- 172.21.15.59:0/2582378119 wait complete. 2024-01-23T08:44:20.749 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi137 2024-01-23T08:44:20.749 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-23T08:44:20.749 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/etc/ceph/ceph.conf 2024-01-23T08:44:20.767 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-23T08:44:20.768 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-23T08:44:20.825 INFO:tasks.cephadm:Adding host smithi137 to orchestrator... 2024-01-23T08:44:20.825 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph orch host add smithi137 2024-01-23T08:44:21.092 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:44:21.241 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:21 smithi059 ceph-mon[28424]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:44:21.241 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:21 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:21.241 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:21 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:21.241 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:21 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:21.241 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:21 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:21.242 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:21 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:22.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:22 smithi059 ceph-mon[28424]: Deploying daemon node-exporter.smithi059 on smithi059 2024-01-23T08:44:22.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.578+0000 7f97395ef700 1 -- 172.21.15.59:0/2102404048 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9734066710 msgr2=0x7f97340ff620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:22.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.578+0000 7f97395ef700 1 --2- 172.21.15.59:0/2102404048 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9734066710 0x7f97340ff620 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f97240048c0 tx=0x7f9724009e30 comp rx=0 tx=0).stop 2024-01-23T08:44:22.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.578+0000 7f97395ef700 1 -- 172.21.15.59:0/2102404048 shutdown_connections 2024-01-23T08:44:22.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.578+0000 7f97395ef700 1 --2- 172.21.15.59:0/2102404048 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9734066710 0x7f97340ff620 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:22.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.578+0000 7f97395ef700 1 -- 172.21.15.59:0/2102404048 >> 172.21.15.59:0/2102404048 conn(0x7f97340f7030 msgr2=0x7f97340f9440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:22.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.578+0000 7f97395ef700 1 -- 172.21.15.59:0/2102404048 shutdown_connections 2024-01-23T08:44:22.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.578+0000 7f97395ef700 1 -- 172.21.15.59:0/2102404048 wait complete. 2024-01-23T08:44:22.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.579+0000 7f97395ef700 1 Processor -- start 2024-01-23T08:44:22.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.579+0000 7f97395ef700 1 -- start start 2024-01-23T08:44:22.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.579+0000 7f97395ef700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9734066710 0x7f97340fd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:22.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.579+0000 7f97395ef700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97340fb520 con 0x7f9734066710 2024-01-23T08:44:22.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.579+0000 7f9732ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9734066710 0x7f97340fd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:22.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.579+0000 7f9732ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9734066710 0x7f97340fd0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36678/0 (socket says 172.21.15.59:36678) 2024-01-23T08:44:22.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.579+0000 7f9732ffd700 1 -- 172.21.15.59:0/3686546534 learned_addr learned my addr 172.21.15.59:0/3686546534 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:44:22.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.580+0000 7f9732ffd700 1 -- 172.21.15.59:0/3686546534 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9724005040 con 0x7f9734066710 2024-01-23T08:44:22.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.580+0000 7f9732ffd700 1 --2- 172.21.15.59:0/3686546534 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9734066710 0x7f97340fd0a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9724009040 tx=0x7f9724009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:22.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.580+0000 7f971b7fe700 1 -- 172.21.15.59:0/3686546534 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9724009620 con 0x7f9734066710 2024-01-23T08:44:22.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.580+0000 7f971b7fe700 1 -- 172.21.15.59:0/3686546534 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9724009780 con 0x7f9734066710 2024-01-23T08:44:22.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.580+0000 7f97395ef700 1 -- 172.21.15.59:0/3686546534 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97340fb720 con 0x7f9734066710 2024-01-23T08:44:22.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.580+0000 7f971b7fe700 1 -- 172.21.15.59:0/3686546534 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f972401c5b0 con 0x7f9734066710 2024-01-23T08:44:22.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.580+0000 7f97395ef700 1 -- 172.21.15.59:0/3686546534 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97340fbc10 con 0x7f9734066710 2024-01-23T08:44:22.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.581+0000 7f971b7fe700 1 -- 172.21.15.59:0/3686546534 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f972401c820 con 0x7f9734066710 2024-01-23T08:44:22.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.581+0000 7f97395ef700 1 -- 172.21.15.59:0/3686546534 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97341066d0 con 0x7f9734066710 2024-01-23T08:44:22.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.581+0000 7f971b7fe700 1 --2- 172.21.15.59:0/3686546534 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f971c036720 0x7f971c038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:22.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.581+0000 7f971b7fe700 1 -- 172.21.15.59:0/3686546534 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9724049900 con 0x7f9734066710 2024-01-23T08:44:22.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.581+0000 7f97327fc700 1 --2- 172.21.15.59:0/3686546534 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f971c036720 0x7f971c038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:22.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.582+0000 7f97327fc700 1 --2- 172.21.15.59:0/3686546534 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f971c036720 0x7f971c038bd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9728009990 tx=0x7f9728006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:22.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.586+0000 7f971b7fe700 1 -- 172.21.15.59:0/3686546534 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f9724023300 con 0x7f9734066710 2024-01-23T08:44:22.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:22.790+0000 7f97395ef700 1 -- 172.21.15.59:0/3686546534 --> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi137", "target": ["mon-mgr", ""]}) v1 -- 0x7f973402b340 con 0x7f971c036720 2024-01-23T08:44:23.428 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:23 smithi059 ceph-mon[28424]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi137", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:44:24.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:24.875+0000 7f971b7fe700 1 -- 172.21.15.59:0/3686546534 <== mgr.14162 v2:172.21.15.59:6800/3614124404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f973402b340 con 0x7f971c036720 2024-01-23T08:44:24.874 INFO:teuthology.orchestra.run.smithi059.stdout:Added host 'smithi137' with addr '172.21.15.137' 2024-01-23T08:44:24.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:24.877+0000 7f97395ef700 1 -- 172.21.15.59:0/3686546534 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f971c036720 msgr2=0x7f971c038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:24.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:24.877+0000 7f97395ef700 1 --2- 172.21.15.59:0/3686546534 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f971c036720 0x7f971c038bd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9728009990 tx=0x7f9728006e40 comp rx=0 tx=0).stop 2024-01-23T08:44:24.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:24.877+0000 7f97395ef700 1 -- 172.21.15.59:0/3686546534 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9734066710 msgr2=0x7f97340fd0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:24.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:24.877+0000 7f97395ef700 1 --2- 172.21.15.59:0/3686546534 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9734066710 0x7f97340fd0a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9724009040 tx=0x7f9724009e30 comp rx=0 tx=0).stop 2024-01-23T08:44:24.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:24.877+0000 7f97395ef700 1 -- 172.21.15.59:0/3686546534 shutdown_connections 2024-01-23T08:44:24.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:24.877+0000 7f97395ef700 1 --2- 172.21.15.59:0/3686546534 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f971c036720 0x7f971c038bd0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:24.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:24.877+0000 7f97395ef700 1 --2- 172.21.15.59:0/3686546534 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9734066710 0x7f97340fd0a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:24.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:24.877+0000 7f97395ef700 1 -- 172.21.15.59:0/3686546534 >> 172.21.15.59:0/3686546534 conn(0x7f97340f7030 msgr2=0x7f97340f7d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:24.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:24.877+0000 7f97395ef700 1 -- 172.21.15.59:0/3686546534 shutdown_connections 2024-01-23T08:44:24.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:24.877+0000 7f97395ef700 1 -- 172.21.15.59:0/3686546534 wait complete. 2024-01-23T08:44:25.099 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:24 smithi059 ceph-mon[28424]: Deploying cephadm binary to smithi137 2024-01-23T08:44:25.100 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:24 smithi059 ceph-mon[28424]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:44:25.100 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:24 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:25.820 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph orch host ls --format=json 2024-01-23T08:44:26.054 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:25 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:26.055 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:25 smithi059 ceph-mon[28424]: Added host smithi137 2024-01-23T08:44:26.236 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:44:27.324 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:26 smithi059 ceph-mon[28424]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:44:27.324 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:26 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:27.324 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:26 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:27.324 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:26 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:27.324 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:26 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:28.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.844+0000 7f3bfa4da700 1 -- 172.21.15.59:0/1302053603 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bf40fe150 msgr2=0x7f3bf40fe560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:28.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.844+0000 7f3bfa4da700 1 --2- 172.21.15.59:0/1302053603 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bf40fe150 0x7f3bf40fe560 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3be40048c0 tx=0x7f3be4009e30 comp rx=0 tx=0).stop 2024-01-23T08:44:28.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.844+0000 7f3bfa4da700 1 -- 172.21.15.59:0/1302053603 shutdown_connections 2024-01-23T08:44:28.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.844+0000 7f3bfa4da700 1 --2- 172.21.15.59:0/1302053603 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bf40fe150 0x7f3bf40fe560 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:28.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.844+0000 7f3bfa4da700 1 -- 172.21.15.59:0/1302053603 >> 172.21.15.59:0/1302053603 conn(0x7f3bf40f6850 msgr2=0x7f3bf40f8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:28.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.845+0000 7f3bfa4da700 1 -- 172.21.15.59:0/1302053603 shutdown_connections 2024-01-23T08:44:28.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.845+0000 7f3bfa4da700 1 -- 172.21.15.59:0/1302053603 wait complete. 2024-01-23T08:44:28.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.845+0000 7f3bfa4da700 1 Processor -- start 2024-01-23T08:44:28.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.845+0000 7f3bfa4da700 1 -- start start 2024-01-23T08:44:28.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.845+0000 7f3bfa4da700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bf40fe150 0x7f3bf410cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:28.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.845+0000 7f3bfa4da700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf410d290 con 0x7f3bf40fe150 2024-01-23T08:44:28.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.845+0000 7f3bf3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bf40fe150 0x7f3bf410cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:28.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.845+0000 7f3bf3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bf40fe150 0x7f3bf410cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37976/0 (socket says 172.21.15.59:37976) 2024-01-23T08:44:28.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.845+0000 7f3bf3fff700 1 -- 172.21.15.59:0/1452529919 learned_addr learned my addr 172.21.15.59:0/1452529919 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:44:28.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.846+0000 7f3bf3fff700 1 -- 172.21.15.59:0/1452529919 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be4005040 con 0x7f3bf40fe150 2024-01-23T08:44:28.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.846+0000 7f3bf3fff700 1 --2- 172.21.15.59:0/1452529919 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bf40fe150 0x7f3bf410cd50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3be4009040 tx=0x7f3be4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:28.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.846+0000 7f3bf0ff9700 1 -- 172.21.15.59:0/1452529919 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3be4004e60 con 0x7f3bf40fe150 2024-01-23T08:44:28.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.846+0000 7f3bf0ff9700 1 -- 172.21.15.59:0/1452529919 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3be4009650 con 0x7f3bf40fe150 2024-01-23T08:44:28.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.846+0000 7f3bfa4da700 1 -- 172.21.15.59:0/1452529919 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf410d490 con 0x7f3bf40fe150 2024-01-23T08:44:28.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.846+0000 7f3bf0ff9700 1 -- 172.21.15.59:0/1452529919 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3be401c3f0 con 0x7f3bf40fe150 2024-01-23T08:44:28.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.846+0000 7f3bfa4da700 1 -- 172.21.15.59:0/1452529919 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf410e050 con 0x7f3bf40fe150 2024-01-23T08:44:28.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.847+0000 7f3bf0ff9700 1 -- 172.21.15.59:0/1452529919 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f3be401c670 con 0x7f3bf40fe150 2024-01-23T08:44:28.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.847+0000 7f3bfa4da700 1 -- 172.21.15.59:0/1452529919 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf405f450 con 0x7f3bf40fe150 2024-01-23T08:44:28.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.847+0000 7f3bf0ff9700 1 --2- 172.21.15.59:0/1452529919 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f3bdc0366d0 0x7f3bdc038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:28.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.847+0000 7f3bf0ff9700 1 -- 172.21.15.59:0/1452529919 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3be4049820 con 0x7f3bf40fe150 2024-01-23T08:44:28.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.847+0000 7f3bf37fe700 1 --2- 172.21.15.59:0/1452529919 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f3bdc0366d0 0x7f3bdc038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:28.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.847+0000 7f3bf37fe700 1 --2- 172.21.15.59:0/1452529919 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f3bdc0366d0 0x7f3bdc038b80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3be8009990 tx=0x7f3be8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:28.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:27.852+0000 7f3bf0ff9700 1 -- 172.21.15.59:0/1452529919 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3be4023300 con 0x7f3bf40fe150 2024-01-23T08:44:28.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:28.060+0000 7f3bfa4da700 1 -- 172.21.15.59:0/1452529919 --> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3bf410ed00 con 0x7f3bdc0366d0 2024-01-23T08:44:28.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:28.061+0000 7f3bf0ff9700 1 -- 172.21.15.59:0/1452529919 <== mgr.14162 v2:172.21.15.59:6800/3614124404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f3bf410ed00 con 0x7f3bdc0366d0 2024-01-23T08:44:28.059 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:44:28.060 INFO:teuthology.orchestra.run.smithi059.stdout:[{"addr": "172.21.15.59", "hostname": "smithi059", "labels": [], "status": ""}, {"addr": "172.21.15.137", "hostname": "smithi137", "labels": [], "status": ""}] 2024-01-23T08:44:28.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:28.063+0000 7f3bfa4da700 1 -- 172.21.15.59:0/1452529919 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f3bdc0366d0 msgr2=0x7f3bdc038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:28.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:28.063+0000 7f3bfa4da700 1 --2- 172.21.15.59:0/1452529919 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f3bdc0366d0 0x7f3bdc038b80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3be8009990 tx=0x7f3be8006e40 comp rx=0 tx=0).stop 2024-01-23T08:44:28.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:28.063+0000 7f3bfa4da700 1 -- 172.21.15.59:0/1452529919 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bf40fe150 msgr2=0x7f3bf410cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:28.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:28.063+0000 7f3bfa4da700 1 --2- 172.21.15.59:0/1452529919 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bf40fe150 0x7f3bf410cd50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3be4009040 tx=0x7f3be4009e30 comp rx=0 tx=0).stop 2024-01-23T08:44:28.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:28.063+0000 7f3bfa4da700 1 -- 172.21.15.59:0/1452529919 shutdown_connections 2024-01-23T08:44:28.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:28.063+0000 7f3bfa4da700 1 --2- 172.21.15.59:0/1452529919 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f3bdc0366d0 0x7f3bdc038b80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:28.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:28.063+0000 7f3bfa4da700 1 --2- 172.21.15.59:0/1452529919 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bf40fe150 0x7f3bf410cd50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:28.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:28.063+0000 7f3bfa4da700 1 -- 172.21.15.59:0/1452529919 >> 172.21.15.59:0/1452529919 conn(0x7f3bf40f6850 msgr2=0x7f3bf40f7660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:28.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:28.063+0000 7f3bfa4da700 1 -- 172.21.15.59:0/1452529919 shutdown_connections 2024-01-23T08:44:28.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:28.063+0000 7f3bfa4da700 1 -- 172.21.15.59:0/1452529919 wait complete. 2024-01-23T08:44:28.460 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:27 smithi059 ceph-mon[28424]: Deploying daemon alertmanager.smithi059 on smithi059 2024-01-23T08:44:28.986 INFO:tasks.cephadm:Setting crush tunables to default 2024-01-23T08:44:28.986 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph osd crush tunables default 2024-01-23T08:44:29.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:28 smithi059 ceph-mon[28424]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:44:29.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:28 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:29.244 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:44:30.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:29 smithi059 ceph-mon[28424]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-23T08:44:31.371 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:30 smithi059 ceph-mon[28424]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:44:32.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.971+0000 7f9501ea5700 1 -- 172.21.15.59:0/1891644701 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94fc0fca50 msgr2=0x7f94fc0fce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:32.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.971+0000 7f9501ea5700 1 --2- 172.21.15.59:0/1891644701 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94fc0fca50 0x7f94fc0fce60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f94ec0048c0 tx=0x7f94ec009e30 comp rx=0 tx=0).stop 2024-01-23T08:44:32.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.972+0000 7f9501ea5700 1 -- 172.21.15.59:0/1891644701 shutdown_connections 2024-01-23T08:44:32.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.972+0000 7f9501ea5700 1 --2- 172.21.15.59:0/1891644701 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94fc0fca50 0x7f94fc0fce60 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:32.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.972+0000 7f9501ea5700 1 -- 172.21.15.59:0/1891644701 >> 172.21.15.59:0/1891644701 conn(0x7f94fc0f8130 msgr2=0x7f94fc0fa540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:32.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.972+0000 7f9501ea5700 1 -- 172.21.15.59:0/1891644701 shutdown_connections 2024-01-23T08:44:32.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.972+0000 7f9501ea5700 1 -- 172.21.15.59:0/1891644701 wait complete. 2024-01-23T08:44:32.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.972+0000 7f9501ea5700 1 Processor -- start 2024-01-23T08:44:32.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.972+0000 7f9501ea5700 1 -- start start 2024-01-23T08:44:32.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.972+0000 7f9501ea5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94fc0fca50 0x7f94fc10cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:32.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.972+0000 7f9501ea5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94fc10d240 con 0x7f94fc0fca50 2024-01-23T08:44:32.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.973+0000 7f94fb7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94fc0fca50 0x7f94fc10cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:32.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.973+0000 7f94fb7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94fc0fca50 0x7f94fc10cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37998/0 (socket says 172.21.15.59:37998) 2024-01-23T08:44:32.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.973+0000 7f94fb7fe700 1 -- 172.21.15.59:0/3557734549 learned_addr learned my addr 172.21.15.59:0/3557734549 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:44:32.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.973+0000 7f94fb7fe700 1 -- 172.21.15.59:0/3557734549 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ec005040 con 0x7f94fc0fca50 2024-01-23T08:44:32.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.973+0000 7f94fb7fe700 1 --2- 172.21.15.59:0/3557734549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94fc0fca50 0x7f94fc10cd00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f94ec00b690 tx=0x7f94ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:32.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.973+0000 7f94e3fff700 1 -- 172.21.15.59:0/3557734549 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94ec009650 con 0x7f94fc0fca50 2024-01-23T08:44:32.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.973+0000 7f9501ea5700 1 -- 172.21.15.59:0/3557734549 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94fc10d440 con 0x7f94fc0fca50 2024-01-23T08:44:32.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.973+0000 7f94e3fff700 1 -- 172.21.15.59:0/3557734549 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f94ec0097b0 con 0x7f94fc0fca50 2024-01-23T08:44:32.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.973+0000 7f94e3fff700 1 -- 172.21.15.59:0/3557734549 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94ec01c5d0 con 0x7f94fc0fca50 2024-01-23T08:44:32.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.973+0000 7f9501ea5700 1 -- 172.21.15.59:0/3557734549 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94fc10dfa0 con 0x7f94fc0fca50 2024-01-23T08:44:32.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.974+0000 7f94e3fff700 1 -- 172.21.15.59:0/3557734549 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f94ec01c840 con 0x7f94fc0fca50 2024-01-23T08:44:32.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.974+0000 7f9501ea5700 1 -- 172.21.15.59:0/3557734549 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94fc04bd60 con 0x7f94fc0fca50 2024-01-23T08:44:32.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.974+0000 7f94e3fff700 1 --2- 172.21.15.59:0/3557734549 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f94e40366d0 0x7f94e4038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:32.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.974+0000 7f94e3fff700 1 -- 172.21.15.59:0/3557734549 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f94ec049a50 con 0x7f94fc0fca50 2024-01-23T08:44:32.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.974+0000 7f94faffd700 1 --2- 172.21.15.59:0/3557734549 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f94e40366d0 0x7f94e4038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:32.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.975+0000 7f94faffd700 1 --2- 172.21.15.59:0/3557734549 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f94e40366d0 0x7f94e4038b80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f94f0009990 tx=0x7f94f0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:32.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:31.979+0000 7f94e3fff700 1 -- 172.21.15.59:0/3557734549 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f94ec011550 con 0x7f94fc0fca50 2024-01-23T08:44:32.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:32.175+0000 7f9501ea5700 1 -- 172.21.15.59:0/3557734549 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f94fc106100 con 0x7f94fc0fca50 2024-01-23T08:44:32.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:32.987+0000 7f94e3fff700 1 -- 172.21.15.59:0/3557734549 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f94ec011370 con 0x7f94fc0fca50 2024-01-23T08:44:32.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:32.989+0000 7f9501ea5700 1 -- 172.21.15.59:0/3557734549 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f94e40366d0 msgr2=0x7f94e4038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:32.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:32.989+0000 7f9501ea5700 1 --2- 172.21.15.59:0/3557734549 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f94e40366d0 0x7f94e4038b80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f94f0009990 tx=0x7f94f0006e40 comp rx=0 tx=0).stop 2024-01-23T08:44:32.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:32.989+0000 7f9501ea5700 1 -- 172.21.15.59:0/3557734549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94fc0fca50 msgr2=0x7f94fc10cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:32.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:32.989+0000 7f9501ea5700 1 --2- 172.21.15.59:0/3557734549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94fc0fca50 0x7f94fc10cd00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f94ec00b690 tx=0x7f94ec009e30 comp rx=0 tx=0).stop 2024-01-23T08:44:32.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:32.989+0000 7f9501ea5700 1 -- 172.21.15.59:0/3557734549 shutdown_connections 2024-01-23T08:44:32.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:32.989+0000 7f9501ea5700 1 --2- 172.21.15.59:0/3557734549 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f94e40366d0 0x7f94e4038b80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:32.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:32.989+0000 7f9501ea5700 1 --2- 172.21.15.59:0/3557734549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94fc0fca50 0x7f94fc10cd00 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:32.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:32.989+0000 7f9501ea5700 1 -- 172.21.15.59:0/3557734549 >> 172.21.15.59:0/3557734549 conn(0x7f94fc0f8130 msgr2=0x7f94fc0f8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:32.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:32.990+0000 7f9501ea5700 1 -- 172.21.15.59:0/3557734549 shutdown_connections 2024-01-23T08:44:32.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:44:32.990+0000 7f9501ea5700 1 -- 172.21.15.59:0/3557734549 wait complete. 2024-01-23T08:44:32.989 INFO:teuthology.orchestra.run.smithi059.stderr:adjusted tunables profile to default 2024-01-23T08:44:33.194 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:32 smithi059 ceph-mon[28424]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:44:33.195 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:32 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/3557734549' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-23T08:44:34.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:33 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/3557734549' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-23T08:44:34.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:33 smithi059 ceph-mon[28424]: osdmap e4: 0 total, 0 up, 0 in 2024-01-23T08:44:34.332 INFO:tasks.cephadm:Adding mon.smithi059 on smithi059 2024-01-23T08:44:34.332 INFO:tasks.cephadm:Adding mon.smithi137 on smithi137 2024-01-23T08:44:34.332 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph orch apply mon '2;smithi059:172.21.15.59=smithi059;smithi137:172.21.15.137=smithi137' 2024-01-23T08:44:34.580 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-23T08:44:34.627 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-23T08:44:35.089 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:34 smithi059 ceph-mon[28424]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:44:36.122 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.124+0000 7f8dc4c6d700 1 -- 172.21.15.137:0/1387328202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dc00fca50 msgr2=0x7f8dc00fce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:36.122 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.124+0000 7f8dc4c6d700 1 --2- 172.21.15.137:0/1387328202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dc00fca50 0x7f8dc00fce60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8db00048c0 tx=0x7f8db0009e30 comp rx=0 tx=0).stop 2024-01-23T08:44:36.122 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.124+0000 7f8dc4c6d700 1 -- 172.21.15.137:0/1387328202 shutdown_connections 2024-01-23T08:44:36.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.124+0000 7f8dc4c6d700 1 --2- 172.21.15.137:0/1387328202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dc00fca50 0x7f8dc00fce60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:36.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.124+0000 7f8dc4c6d700 1 -- 172.21.15.137:0/1387328202 >> 172.21.15.137:0/1387328202 conn(0x7f8dc00f8130 msgr2=0x7f8dc00fa540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:36.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.124+0000 7f8dc4c6d700 1 -- 172.21.15.137:0/1387328202 shutdown_connections 2024-01-23T08:44:36.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.124+0000 7f8dc4c6d700 1 -- 172.21.15.137:0/1387328202 wait complete. 2024-01-23T08:44:36.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.124+0000 7f8dc4c6d700 1 Processor -- start 2024-01-23T08:44:36.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.124+0000 7f8dc4c6d700 1 -- start start 2024-01-23T08:44:36.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.124+0000 7f8dc4c6d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dc00fca50 0x7f8dc010cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:36.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.124+0000 7f8dc4c6d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc010d250 con 0x7f8dc00fca50 2024-01-23T08:44:36.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.125+0000 7f8dbe59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dc00fca50 0x7f8dc010cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:36.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.125+0000 7f8dbe59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dc00fca50 0x7f8dc010cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.137:40128/0 (socket says 172.21.15.137:40128) 2024-01-23T08:44:36.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.125+0000 7f8dbe59c700 1 -- 172.21.15.137:0/610829245 learned_addr learned my addr 172.21.15.137:0/610829245 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-23T08:44:36.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.125+0000 7f8dbe59c700 1 -- 172.21.15.137:0/610829245 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db0005040 con 0x7f8dc00fca50 2024-01-23T08:44:36.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.125+0000 7f8dbe59c700 1 --2- 172.21.15.137:0/610829245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dc00fca50 0x7f8dc010cd10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8db0009330 tx=0x7f8db0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:36.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.125+0000 7f8db6ffd700 1 -- 172.21.15.137:0/610829245 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8db00096b0 con 0x7f8dc00fca50 2024-01-23T08:44:36.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.125+0000 7f8dc4c6d700 1 -- 172.21.15.137:0/610829245 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc010d450 con 0x7f8dc00fca50 2024-01-23T08:44:36.124 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.125+0000 7f8db6ffd700 1 -- 172.21.15.137:0/610829245 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8db0009810 con 0x7f8dc00fca50 2024-01-23T08:44:36.124 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.125+0000 7f8db6ffd700 1 -- 172.21.15.137:0/610829245 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8db001c5b0 con 0x7f8dc00fca50 2024-01-23T08:44:36.124 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.125+0000 7f8dc4c6d700 1 -- 172.21.15.137:0/610829245 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc010dfb0 con 0x7f8dc00fca50 2024-01-23T08:44:36.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.126+0000 7f8db6ffd700 1 -- 172.21.15.137:0/610829245 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f8db001c870 con 0x7f8dc00fca50 2024-01-23T08:44:36.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.126+0000 7f8db6ffd700 1 --2- 172.21.15.137:0/610829245 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f8dac0366d0 0x7f8dac038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:36.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.126+0000 7f8dc4c6d700 1 -- 172.21.15.137:0/610829245 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc004bd60 con 0x7f8dc00fca50 2024-01-23T08:44:36.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.126+0000 7f8db6ffd700 1 -- 172.21.15.137:0/610829245 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8db0049ba0 con 0x7f8dc00fca50 2024-01-23T08:44:36.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.127+0000 7f8dbdd9b700 1 --2- 172.21.15.137:0/610829245 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f8dac0366d0 0x7f8dac038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:36.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.127+0000 7f8dbdd9b700 1 --2- 172.21.15.137:0/610829245 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f8dac0366d0 0x7f8dac038b80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8da8009990 tx=0x7f8da8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:36.128 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.131+0000 7f8db6ffd700 1 -- 172.21.15.137:0/610829245 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f8db003ee60 con 0x7f8dc00fca50 2024-01-23T08:44:36.316 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.319+0000 7f8dc4c6d700 1 -- 172.21.15.137:0/610829245 --> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi059:172.21.15.59=smithi059;smithi137:172.21.15.137=smithi137", "target": ["mon-mgr", ""]}) v1 -- 0x7f8dc010ecf0 con 0x7f8dac0366d0 2024-01-23T08:44:36.322 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.325+0000 7f8db6ffd700 1 -- 172.21.15.137:0/610829245 <== mgr.14162 v2:172.21.15.59:6800/3614124404 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f8dc010ecf0 con 0x7f8dac0366d0 2024-01-23T08:44:36.322 INFO:teuthology.orchestra.run.smithi137.stdout:Scheduled mon update... 2024-01-23T08:44:36.324 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.327+0000 7f8dc4c6d700 1 -- 172.21.15.137:0/610829245 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f8dac0366d0 msgr2=0x7f8dac038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:36.325 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.327+0000 7f8dc4c6d700 1 --2- 172.21.15.137:0/610829245 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f8dac0366d0 0x7f8dac038b80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8da8009990 tx=0x7f8da8006e40 comp rx=0 tx=0).stop 2024-01-23T08:44:36.325 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.327+0000 7f8dc4c6d700 1 -- 172.21.15.137:0/610829245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dc00fca50 msgr2=0x7f8dc010cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:36.325 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.327+0000 7f8dc4c6d700 1 --2- 172.21.15.137:0/610829245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dc00fca50 0x7f8dc010cd10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8db0009330 tx=0x7f8db0009e30 comp rx=0 tx=0).stop 2024-01-23T08:44:36.325 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.327+0000 7f8dc4c6d700 1 -- 172.21.15.137:0/610829245 shutdown_connections 2024-01-23T08:44:36.325 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.327+0000 7f8dc4c6d700 1 --2- 172.21.15.137:0/610829245 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f8dac0366d0 0x7f8dac038b80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:36.325 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.327+0000 7f8dc4c6d700 1 --2- 172.21.15.137:0/610829245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dc00fca50 0x7f8dc010cd10 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:36.325 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.327+0000 7f8dc4c6d700 1 -- 172.21.15.137:0/610829245 >> 172.21.15.137:0/610829245 conn(0x7f8dc00f8130 msgr2=0x7f8dc00f8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:36.325 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.327+0000 7f8dc4c6d700 1 -- 172.21.15.137:0/610829245 shutdown_connections 2024-01-23T08:44:36.325 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:36.327+0000 7f8dc4c6d700 1 -- 172.21.15.137:0/610829245 wait complete. 2024-01-23T08:44:36.922 DEBUG:teuthology.orchestra.run.smithi137:mon.smithi137> sudo journalctl -f -n 0 -u ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@mon.smithi137.service 2024-01-23T08:44:36.925 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-23T08:44:36.925 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph mon dump -f json 2024-01-23T08:44:37.203 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-23T08:44:37.249 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-23T08:44:37.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:36 smithi059 ceph-mon[28424]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:44:37.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:36 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:37.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:36 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:37.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:36 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:37.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:36 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:37.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:36 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:37.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:36 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:37.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:36 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:37.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:36 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-23T08:44:37.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:36 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:38.273 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:37 smithi059 ceph-mon[28424]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi059:172.21.15.59=smithi059;smithi137:172.21.15.137=smithi137", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:44:38.273 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:37 smithi059 ceph-mon[28424]: Saving service mon spec with placement smithi059:172.21.15.59=smithi059;smithi137:172.21.15.137=smithi137;count:2 2024-01-23T08:44:38.273 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:37 smithi059 ceph-mon[28424]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-23T08:44:38.273 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:37 smithi059 ceph-mon[28424]: Deploying daemon grafana.smithi059 on smithi059 2024-01-23T08:44:38.597 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.599+0000 7f09a9e7b700 1 -- 172.21.15.137:0/602944111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09a40fe090 msgr2=0x7f09a40fe4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:38.597 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.599+0000 7f09a9e7b700 1 --2- 172.21.15.137:0/602944111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09a40fe090 0x7f09a40fe4a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f09940048c0 tx=0x7f0994009e30 comp rx=0 tx=0).stop 2024-01-23T08:44:38.598 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.599+0000 7f09a9e7b700 1 -- 172.21.15.137:0/602944111 shutdown_connections 2024-01-23T08:44:38.598 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.599+0000 7f09a9e7b700 1 --2- 172.21.15.137:0/602944111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09a40fe090 0x7f09a40fe4a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:38.598 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.599+0000 7f09a9e7b700 1 -- 172.21.15.137:0/602944111 >> 172.21.15.137:0/602944111 conn(0x7f09a40f6850 msgr2=0x7f09a40f8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:38.598 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.599+0000 7f09a9e7b700 1 -- 172.21.15.137:0/602944111 shutdown_connections 2024-01-23T08:44:38.598 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.599+0000 7f09a9e7b700 1 -- 172.21.15.137:0/602944111 wait complete. 2024-01-23T08:44:38.598 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.599+0000 7f09a9e7b700 1 Processor -- start 2024-01-23T08:44:38.598 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.600+0000 7f09a9e7b700 1 -- start start 2024-01-23T08:44:38.598 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.600+0000 7f09a9e7b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09a40fe090 0x7f09a410cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:38.598 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.600+0000 7f09a9e7b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a410d290 con 0x7f09a40fe090 2024-01-23T08:44:38.598 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.600+0000 7f09a37fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09a40fe090 0x7f09a410cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:38.598 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.600+0000 7f09a37fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09a40fe090 0x7f09a410cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.137:40154/0 (socket says 172.21.15.137:40154) 2024-01-23T08:44:38.598 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.600+0000 7f09a37fe700 1 -- 172.21.15.137:0/3225837322 learned_addr learned my addr 172.21.15.137:0/3225837322 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-23T08:44:38.598 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.600+0000 7f09a37fe700 1 -- 172.21.15.137:0/3225837322 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0994005040 con 0x7f09a40fe090 2024-01-23T08:44:38.598 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.600+0000 7f09a37fe700 1 --2- 172.21.15.137:0/3225837322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09a40fe090 0x7f09a410cd50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0994009680 tx=0x7f0994009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:38.599 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.600+0000 7f098bfff700 1 -- 172.21.15.137:0/3225837322 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f099400b5c0 con 0x7f09a40fe090 2024-01-23T08:44:38.599 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.601+0000 7f098bfff700 1 -- 172.21.15.137:0/3225837322 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0994004e60 con 0x7f09a40fe090 2024-01-23T08:44:38.599 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.601+0000 7f09a9e7b700 1 -- 172.21.15.137:0/3225837322 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a410d490 con 0x7f09a40fe090 2024-01-23T08:44:38.599 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.601+0000 7f098bfff700 1 -- 172.21.15.137:0/3225837322 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f099401c5b0 con 0x7f09a40fe090 2024-01-23T08:44:38.599 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.601+0000 7f09a9e7b700 1 -- 172.21.15.137:0/3225837322 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a410e050 con 0x7f09a40fe090 2024-01-23T08:44:38.600 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.601+0000 7f098bfff700 1 -- 172.21.15.137:0/3225837322 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f0994013420 con 0x7f09a40fe090 2024-01-23T08:44:38.600 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.601+0000 7f09a9e7b700 1 -- 172.21.15.137:0/3225837322 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a405f450 con 0x7f09a40fe090 2024-01-23T08:44:38.600 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.601+0000 7f098bfff700 1 --2- 172.21.15.137:0/3225837322 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f098c0366d0 0x7f098c038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:38.600 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.602+0000 7f098bfff700 1 -- 172.21.15.137:0/3225837322 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0994049840 con 0x7f09a40fe090 2024-01-23T08:44:38.600 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.602+0000 7f09a2ffd700 1 --2- 172.21.15.137:0/3225837322 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f098c0366d0 0x7f098c038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:38.600 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.602+0000 7f09a2ffd700 1 --2- 172.21.15.137:0/3225837322 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f098c0366d0 0x7f098c038b80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0998009990 tx=0x7f0998006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:38.603 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.606+0000 7f098bfff700 1 -- 172.21.15.137:0/3225837322 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0994023300 con 0x7f09a40fe090 2024-01-23T08:44:38.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.863+0000 7f09a9e7b700 1 -- 172.21.15.137:0/3225837322 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09a410ec90 con 0x7f09a40fe090 2024-01-23T08:44:38.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.864+0000 7f098bfff700 1 -- 172.21.15.137:0/3225837322 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f09940116f0 con 0x7f09a40fe090 2024-01-23T08:44:38.861 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:44:38.861 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"3fc07a38-b9cb-11ee-95b1-87774f69a715","modified":"2024-01-23T08:43:01.494526Z","created":"2024-01-23T08:43:01.494526Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-23T08:44:38.863 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.866+0000 7f09a9e7b700 1 -- 172.21.15.137:0/3225837322 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f098c0366d0 msgr2=0x7f098c038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:38.863 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.866+0000 7f09a9e7b700 1 --2- 172.21.15.137:0/3225837322 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f098c0366d0 0x7f098c038b80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0998009990 tx=0x7f0998006e40 comp rx=0 tx=0).stop 2024-01-23T08:44:38.863 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.866+0000 7f09a9e7b700 1 -- 172.21.15.137:0/3225837322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09a40fe090 msgr2=0x7f09a410cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:38.863 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.866+0000 7f09a9e7b700 1 --2- 172.21.15.137:0/3225837322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09a40fe090 0x7f09a410cd50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0994009680 tx=0x7f0994009e30 comp rx=0 tx=0).stop 2024-01-23T08:44:38.863 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.866+0000 7f09a9e7b700 1 -- 172.21.15.137:0/3225837322 shutdown_connections 2024-01-23T08:44:38.863 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.866+0000 7f09a9e7b700 1 --2- 172.21.15.137:0/3225837322 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f098c0366d0 0x7f098c038b80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:38.863 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.866+0000 7f09a9e7b700 1 --2- 172.21.15.137:0/3225837322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09a40fe090 0x7f09a410cd50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:38.863 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.866+0000 7f09a9e7b700 1 -- 172.21.15.137:0/3225837322 >> 172.21.15.137:0/3225837322 conn(0x7f09a40f6850 msgr2=0x7f09a40f7680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:38.864 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.866+0000 7f09a9e7b700 1 -- 172.21.15.137:0/3225837322 shutdown_connections 2024-01-23T08:44:38.864 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:38.866+0000 7f09a9e7b700 1 -- 172.21.15.137:0/3225837322 wait complete. 2024-01-23T08:44:38.864 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-01-23T08:44:39.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:38 smithi059 ceph-mon[28424]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:44:39.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:38 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:39.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:38 smithi059 ceph-mon[28424]: from='client.? 172.21.15.137:0/3225837322' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:44:40.453 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-23T08:44:40.453 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph mon dump -f json 2024-01-23T08:44:40.701 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-23T08:44:40.748 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-23T08:44:41.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:40 smithi059 ceph-mon[28424]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:44:42.260 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.260+0000 7f9754489700 1 -- 172.21.15.137:0/2429783367 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f974c0fcf30 msgr2=0x7f974c0fd340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:42.260 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.260+0000 7f9754489700 1 --2- 172.21.15.137:0/2429783367 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f974c0fcf30 0x7f974c0fd340 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9744009c60 tx=0x7f9744004d10 comp rx=0 tx=0).stop 2024-01-23T08:44:42.260 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.260+0000 7f9754489700 1 -- 172.21.15.137:0/2429783367 shutdown_connections 2024-01-23T08:44:42.260 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.260+0000 7f9754489700 1 --2- 172.21.15.137:0/2429783367 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f974c0fcf30 0x7f974c0fd340 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:42.260 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.260+0000 7f9754489700 1 -- 172.21.15.137:0/2429783367 >> 172.21.15.137:0/2429783367 conn(0x7f974c0f67b0 msgr2=0x7f974c0f8bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:42.260 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.261+0000 7f9754489700 1 -- 172.21.15.137:0/2429783367 shutdown_connections 2024-01-23T08:44:42.260 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.261+0000 7f9754489700 1 -- 172.21.15.137:0/2429783367 wait complete. 2024-01-23T08:44:42.260 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.261+0000 7f9754489700 1 Processor -- start 2024-01-23T08:44:42.261 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.261+0000 7f9754489700 1 -- start start 2024-01-23T08:44:42.261 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.261+0000 7f9754489700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f974c0fcf30 0x7f974c10fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:42.261 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.261+0000 7f9754489700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c0fb7d0 con 0x7f974c0fcf30 2024-01-23T08:44:42.261 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.261+0000 7f9752225700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f974c0fcf30 0x7f974c10fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:42.261 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.261+0000 7f9752225700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f974c0fcf30 0x7f974c10fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.137:40174/0 (socket says 172.21.15.137:40174) 2024-01-23T08:44:42.261 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.261+0000 7f9752225700 1 -- 172.21.15.137:0/757064726 learned_addr learned my addr 172.21.15.137:0/757064726 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-23T08:44:42.261 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.262+0000 7f9752225700 1 -- 172.21.15.137:0/757064726 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9744009940 con 0x7f974c0fcf30 2024-01-23T08:44:42.261 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.262+0000 7f9752225700 1 --2- 172.21.15.137:0/757064726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f974c0fcf30 0x7f974c10fa80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9744000f80 tx=0x7f9744004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:42.261 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.262+0000 7f9742ffd700 1 -- 172.21.15.137:0/757064726 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f974400bec0 con 0x7f974c0fcf30 2024-01-23T08:44:42.261 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.262+0000 7f9742ffd700 1 -- 172.21.15.137:0/757064726 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f974401d8f0 con 0x7f974c0fcf30 2024-01-23T08:44:42.261 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.262+0000 7f9754489700 1 -- 172.21.15.137:0/757064726 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f974c0fb9d0 con 0x7f974c0fcf30 2024-01-23T08:44:42.261 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.262+0000 7f9742ffd700 1 -- 172.21.15.137:0/757064726 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9744012890 con 0x7f974c0fcf30 2024-01-23T08:44:42.261 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.262+0000 7f9754489700 1 -- 172.21.15.137:0/757064726 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f974c110200 con 0x7f974c0fcf30 2024-01-23T08:44:42.262 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.263+0000 7f9742ffd700 1 -- 172.21.15.137:0/757064726 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f9744019040 con 0x7f974c0fcf30 2024-01-23T08:44:42.262 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.263+0000 7f9754489700 1 -- 172.21.15.137:0/757064726 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f974c1055e0 con 0x7f974c0fcf30 2024-01-23T08:44:42.263 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.263+0000 7f9742ffd700 1 --2- 172.21.15.137:0/757064726 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f97380366d0 0x7f9738038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:42.263 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.263+0000 7f9742ffd700 1 -- 172.21.15.137:0/757064726 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f974404a0c0 con 0x7f974c0fcf30 2024-01-23T08:44:42.263 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.263+0000 7f9751a24700 1 --2- 172.21.15.137:0/757064726 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f97380366d0 0x7f9738038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:42.263 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.264+0000 7f9751a24700 1 --2- 172.21.15.137:0/757064726 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f97380366d0 0x7f9738038b80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f973c009990 tx=0x7f973c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:42.266 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.268+0000 7f9742ffd700 1 -- 172.21.15.137:0/757064726 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f974401d440 con 0x7f974c0fcf30 2024-01-23T08:44:42.519 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.521+0000 7f9754489700 1 -- 172.21.15.137:0/757064726 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f974c04bd60 con 0x7f974c0fcf30 2024-01-23T08:44:42.519 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.521+0000 7f9742ffd700 1 -- 172.21.15.137:0/757064726 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f9744021020 con 0x7f974c0fcf30 2024-01-23T08:44:42.519 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:44:42.519 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"3fc07a38-b9cb-11ee-95b1-87774f69a715","modified":"2024-01-23T08:43:01.494526Z","created":"2024-01-23T08:43:01.494526Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-23T08:44:42.521 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.523+0000 7f9754489700 1 -- 172.21.15.137:0/757064726 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f97380366d0 msgr2=0x7f9738038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:42.521 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.523+0000 7f9754489700 1 --2- 172.21.15.137:0/757064726 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f97380366d0 0x7f9738038b80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f973c009990 tx=0x7f973c006e40 comp rx=0 tx=0).stop 2024-01-23T08:44:42.521 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.523+0000 7f9754489700 1 -- 172.21.15.137:0/757064726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f974c0fcf30 msgr2=0x7f974c10fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:42.521 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.523+0000 7f9754489700 1 --2- 172.21.15.137:0/757064726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f974c0fcf30 0x7f974c10fa80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9744000f80 tx=0x7f9744004d10 comp rx=0 tx=0).stop 2024-01-23T08:44:42.521 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.523+0000 7f9754489700 1 -- 172.21.15.137:0/757064726 shutdown_connections 2024-01-23T08:44:42.521 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.523+0000 7f9754489700 1 --2- 172.21.15.137:0/757064726 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f97380366d0 0x7f9738038b80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:42.522 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.523+0000 7f9754489700 1 --2- 172.21.15.137:0/757064726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f974c0fcf30 0x7f974c10fa80 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:42.522 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.523+0000 7f9754489700 1 -- 172.21.15.137:0/757064726 >> 172.21.15.137:0/757064726 conn(0x7f974c0f67b0 msgr2=0x7f974c0f76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:42.522 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.523+0000 7f9754489700 1 -- 172.21.15.137:0/757064726 shutdown_connections 2024-01-23T08:44:42.522 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:42.523+0000 7f9754489700 1 -- 172.21.15.137:0/757064726 wait complete. 2024-01-23T08:44:42.522 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-01-23T08:44:43.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:42 smithi059 ceph-mon[28424]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:44:43.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:42 smithi059 ceph-mon[28424]: from='client.? 172.21.15.137:0/757064726' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:44:44.122 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-23T08:44:44.123 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph mon dump -f json 2024-01-23T08:44:44.378 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-23T08:44:44.430 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-23T08:44:45.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:45 smithi059 ceph-mon[28424]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:44:45.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.796+0000 7fb50342f700 1 -- 172.21.15.137:0/1386802268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4fc0fad30 msgr2=0x7fb4fc0fb140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:45.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.796+0000 7fb50342f700 1 --2- 172.21.15.137:0/1386802268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4fc0fad30 0x7fb4fc0fb140 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb4f40048c0 tx=0x7fb4f4009e30 comp rx=0 tx=0).stop 2024-01-23T08:44:45.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.796+0000 7fb50342f700 1 -- 172.21.15.137:0/1386802268 shutdown_connections 2024-01-23T08:44:45.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.797+0000 7fb50342f700 1 --2- 172.21.15.137:0/1386802268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4fc0fad30 0x7fb4fc0fb140 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:45.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.797+0000 7fb50342f700 1 -- 172.21.15.137:0/1386802268 >> 172.21.15.137:0/1386802268 conn(0x7fb4fc0f67b0 msgr2=0x7fb4fc0f8be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:45.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.797+0000 7fb50342f700 1 -- 172.21.15.137:0/1386802268 shutdown_connections 2024-01-23T08:44:45.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.797+0000 7fb50342f700 1 -- 172.21.15.137:0/1386802268 wait complete. 2024-01-23T08:44:45.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.797+0000 7fb50342f700 1 Processor -- start 2024-01-23T08:44:45.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.797+0000 7fb50342f700 1 -- start start 2024-01-23T08:44:45.796 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.797+0000 7fb50342f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4fc0fad30 0x7fb4fc10cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:45.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.797+0000 7fb50342f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc10d280 con 0x7fb4fc0fad30 2024-01-23T08:44:45.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.798+0000 7fb5011cb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4fc0fad30 0x7fb4fc10cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:45.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.798+0000 7fb5011cb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4fc0fad30 0x7fb4fc10cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.137:49016/0 (socket says 172.21.15.137:49016) 2024-01-23T08:44:45.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.798+0000 7fb5011cb700 1 -- 172.21.15.137:0/4245338094 learned_addr learned my addr 172.21.15.137:0/4245338094 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-23T08:44:45.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.798+0000 7fb5011cb700 1 -- 172.21.15.137:0/4245338094 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f4005040 con 0x7fb4fc0fad30 2024-01-23T08:44:45.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.798+0000 7fb5011cb700 1 --2- 172.21.15.137:0/4245338094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4fc0fad30 0x7fb4fc10cd40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4009300 tx=0x7fb4f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:45.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.798+0000 7fb4f9ffb700 1 -- 172.21.15.137:0/4245338094 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4f4004e60 con 0x7fb4fc0fad30 2024-01-23T08:44:45.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.798+0000 7fb4f9ffb700 1 -- 172.21.15.137:0/4245338094 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4f4009680 con 0x7fb4fc0fad30 2024-01-23T08:44:45.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.798+0000 7fb50342f700 1 -- 172.21.15.137:0/4245338094 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4fc10d480 con 0x7fb4fc0fad30 2024-01-23T08:44:45.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.798+0000 7fb4f9ffb700 1 -- 172.21.15.137:0/4245338094 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4f401c5b0 con 0x7fb4fc0fad30 2024-01-23T08:44:45.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.799+0000 7fb50342f700 1 -- 172.21.15.137:0/4245338094 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4fc10dfe0 con 0x7fb4fc0fad30 2024-01-23T08:44:45.798 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.799+0000 7fb50342f700 1 -- 172.21.15.137:0/4245338094 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4fc04bd60 con 0x7fb4fc0fad30 2024-01-23T08:44:45.798 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.799+0000 7fb4f9ffb700 1 -- 172.21.15.137:0/4245338094 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fb4f4013420 con 0x7fb4fc0fad30 2024-01-23T08:44:45.798 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.800+0000 7fb4f9ffb700 1 --2- 172.21.15.137:0/4245338094 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fb4e40366d0 0x7fb4e4038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:45.798 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.800+0000 7fb4f9ffb700 1 -- 172.21.15.137:0/4245338094 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb4f4049910 con 0x7fb4fc0fad30 2024-01-23T08:44:45.799 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.800+0000 7fb5009ca700 1 --2- 172.21.15.137:0/4245338094 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fb4e40366d0 0x7fb4e4038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:45.799 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.801+0000 7fb5009ca700 1 --2- 172.21.15.137:0/4245338094 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fb4e40366d0 0x7fb4e4038b80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0009990 tx=0x7fb4f0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:45.802 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:45.805+0000 7fb4f9ffb700 1 -- 172.21.15.137:0/4245338094 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb4f4011550 con 0x7fb4fc0fad30 2024-01-23T08:44:46.055 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:46.057+0000 7fb50342f700 1 -- 172.21.15.137:0/4245338094 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb4fc10ee60 con 0x7fb4fc0fad30 2024-01-23T08:44:46.056 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:46.058+0000 7fb4f9ffb700 1 -- 172.21.15.137:0/4245338094 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fb4f4011370 con 0x7fb4fc0fad30 2024-01-23T08:44:46.056 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:44:46.056 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"3fc07a38-b9cb-11ee-95b1-87774f69a715","modified":"2024-01-23T08:43:01.494526Z","created":"2024-01-23T08:43:01.494526Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-23T08:44:46.058 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:46.060+0000 7fb50342f700 1 -- 172.21.15.137:0/4245338094 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fb4e40366d0 msgr2=0x7fb4e4038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:46.058 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:46.060+0000 7fb50342f700 1 --2- 172.21.15.137:0/4245338094 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fb4e40366d0 0x7fb4e4038b80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0009990 tx=0x7fb4f0006e40 comp rx=0 tx=0).stop 2024-01-23T08:44:46.058 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:46.060+0000 7fb50342f700 1 -- 172.21.15.137:0/4245338094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4fc0fad30 msgr2=0x7fb4fc10cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:46.058 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:46.060+0000 7fb50342f700 1 --2- 172.21.15.137:0/4245338094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4fc0fad30 0x7fb4fc10cd40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4009300 tx=0x7fb4f4009e30 comp rx=0 tx=0).stop 2024-01-23T08:44:46.058 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:46.060+0000 7fb50342f700 1 -- 172.21.15.137:0/4245338094 shutdown_connections 2024-01-23T08:44:46.058 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:46.060+0000 7fb50342f700 1 --2- 172.21.15.137:0/4245338094 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fb4e40366d0 0x7fb4e4038b80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:46.058 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:46.060+0000 7fb50342f700 1 --2- 172.21.15.137:0/4245338094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4fc0fad30 0x7fb4fc10cd40 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:46.058 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:46.060+0000 7fb50342f700 1 -- 172.21.15.137:0/4245338094 >> 172.21.15.137:0/4245338094 conn(0x7fb4fc0f67b0 msgr2=0x7fb4fc0f75a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:46.058 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:46.060+0000 7fb50342f700 1 -- 172.21.15.137:0/4245338094 shutdown_connections 2024-01-23T08:44:46.058 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:46.060+0000 7fb50342f700 1 -- 172.21.15.137:0/4245338094 wait complete. 2024-01-23T08:44:46.059 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-01-23T08:44:47.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:47 smithi059 ceph-mon[28424]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:44:47.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:47 smithi059 ceph-mon[28424]: from='client.? 172.21.15.137:0/4245338094' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:44:47.761 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-23T08:44:47.762 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph mon dump -f json 2024-01-23T08:44:48.005 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-23T08:44:48.052 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-23T08:44:49.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:49 smithi059 ceph-mon[28424]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:44:49.480 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.482+0000 7f9f751a7700 1 -- 172.21.15.137:0/4244151002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f700fad10 msgr2=0x7f9f700fb120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:49.481 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.482+0000 7f9f751a7700 1 --2- 172.21.15.137:0/4244151002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f700fad10 0x7f9f700fb120 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9f600048c0 tx=0x7f9f60009e30 comp rx=0 tx=0).stop 2024-01-23T08:44:49.481 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.483+0000 7f9f751a7700 1 -- 172.21.15.137:0/4244151002 shutdown_connections 2024-01-23T08:44:49.481 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.483+0000 7f9f751a7700 1 --2- 172.21.15.137:0/4244151002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f700fad10 0x7f9f700fb120 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:49.481 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.483+0000 7f9f751a7700 1 -- 172.21.15.137:0/4244151002 >> 172.21.15.137:0/4244151002 conn(0x7f9f700f67b0 msgr2=0x7f9f700f8bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:49.481 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.483+0000 7f9f751a7700 1 -- 172.21.15.137:0/4244151002 shutdown_connections 2024-01-23T08:44:49.481 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.483+0000 7f9f751a7700 1 -- 172.21.15.137:0/4244151002 wait complete. 2024-01-23T08:44:49.483 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.483+0000 7f9f751a7700 1 Processor -- start 2024-01-23T08:44:49.483 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.483+0000 7f9f751a7700 1 -- start start 2024-01-23T08:44:49.483 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.483+0000 7f9f751a7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f700fad10 0x7f9f7010cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:49.483 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.483+0000 7f9f751a7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7010d270 con 0x7f9f700fad10 2024-01-23T08:44:49.483 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.484+0000 7f9f6ed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f700fad10 0x7f9f7010cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:49.483 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.484+0000 7f9f6ed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f700fad10 0x7f9f7010cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.137:49026/0 (socket says 172.21.15.137:49026) 2024-01-23T08:44:49.483 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.484+0000 7f9f6ed9d700 1 -- 172.21.15.137:0/4244508471 learned_addr learned my addr 172.21.15.137:0/4244508471 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-23T08:44:49.483 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.484+0000 7f9f6ed9d700 1 -- 172.21.15.137:0/4244508471 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f60005040 con 0x7f9f700fad10 2024-01-23T08:44:49.483 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.484+0000 7f9f6ed9d700 1 --2- 172.21.15.137:0/4244508471 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f700fad10 0x7f9f7010cd30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9f60009040 tx=0x7f9f60009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:49.484 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.484+0000 7f9f577fe700 1 -- 172.21.15.137:0/4244508471 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f60009650 con 0x7f9f700fad10 2024-01-23T08:44:49.484 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.484+0000 7f9f577fe700 1 -- 172.21.15.137:0/4244508471 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9f600097b0 con 0x7f9f700fad10 2024-01-23T08:44:49.484 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.484+0000 7f9f577fe700 1 -- 172.21.15.137:0/4244508471 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f6001c5b0 con 0x7f9f700fad10 2024-01-23T08:44:49.484 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.484+0000 7f9f751a7700 1 -- 172.21.15.137:0/4244508471 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f7010d470 con 0x7f9f700fad10 2024-01-23T08:44:49.484 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.484+0000 7f9f751a7700 1 -- 172.21.15.137:0/4244508471 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f7010dfd0 con 0x7f9f700fad10 2024-01-23T08:44:49.484 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.485+0000 7f9f577fe700 1 -- 172.21.15.137:0/4244508471 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f9f60013420 con 0x7f9f700fad10 2024-01-23T08:44:49.484 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.485+0000 7f9f751a7700 1 -- 172.21.15.137:0/4244508471 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f7005f450 con 0x7f9f700fad10 2024-01-23T08:44:49.484 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.485+0000 7f9f577fe700 1 --2- 172.21.15.137:0/4244508471 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f9f580366d0 0x7f9f58038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:49.484 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.485+0000 7f9f577fe700 1 -- 172.21.15.137:0/4244508471 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9f600498b0 con 0x7f9f700fad10 2024-01-23T08:44:49.484 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.485+0000 7f9f6e59c700 1 --2- 172.21.15.137:0/4244508471 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f9f580366d0 0x7f9f58038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:49.485 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.486+0000 7f9f6e59c700 1 --2- 172.21.15.137:0/4244508471 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f9f580366d0 0x7f9f58038b80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9f64009990 tx=0x7f9f64006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:49.488 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.490+0000 7f9f577fe700 1 -- 172.21.15.137:0/4244508471 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f9f600139d0 con 0x7f9f700fad10 2024-01-23T08:44:49.742 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.744+0000 7f9f751a7700 1 -- 172.21.15.137:0/4244508471 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9f7004bd60 con 0x7f9f700fad10 2024-01-23T08:44:49.743 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.745+0000 7f9f577fe700 1 -- 172.21.15.137:0/4244508471 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f9f60011370 con 0x7f9f700fad10 2024-01-23T08:44:49.743 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:44:49.743 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"3fc07a38-b9cb-11ee-95b1-87774f69a715","modified":"2024-01-23T08:43:01.494526Z","created":"2024-01-23T08:43:01.494526Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-23T08:44:49.745 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.747+0000 7f9f751a7700 1 -- 172.21.15.137:0/4244508471 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f9f580366d0 msgr2=0x7f9f58038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:49.745 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.747+0000 7f9f751a7700 1 --2- 172.21.15.137:0/4244508471 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f9f580366d0 0x7f9f58038b80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9f64009990 tx=0x7f9f64006e40 comp rx=0 tx=0).stop 2024-01-23T08:44:49.745 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.747+0000 7f9f751a7700 1 -- 172.21.15.137:0/4244508471 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f700fad10 msgr2=0x7f9f7010cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:49.745 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.747+0000 7f9f751a7700 1 --2- 172.21.15.137:0/4244508471 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f700fad10 0x7f9f7010cd30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9f60009040 tx=0x7f9f60009e30 comp rx=0 tx=0).stop 2024-01-23T08:44:49.745 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.747+0000 7f9f751a7700 1 -- 172.21.15.137:0/4244508471 shutdown_connections 2024-01-23T08:44:49.745 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.747+0000 7f9f751a7700 1 --2- 172.21.15.137:0/4244508471 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f9f580366d0 0x7f9f58038b80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:49.745 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.747+0000 7f9f751a7700 1 --2- 172.21.15.137:0/4244508471 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f700fad10 0x7f9f7010cd30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:49.745 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.747+0000 7f9f751a7700 1 -- 172.21.15.137:0/4244508471 >> 172.21.15.137:0/4244508471 conn(0x7f9f700f67b0 msgr2=0x7f9f700f7600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:49.745 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.747+0000 7f9f751a7700 1 -- 172.21.15.137:0/4244508471 shutdown_connections 2024-01-23T08:44:49.745 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:49.747+0000 7f9f751a7700 1 -- 172.21.15.137:0/4244508471 wait complete. 2024-01-23T08:44:49.746 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-01-23T08:44:50.318 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:50 smithi059 ceph-mon[28424]: from='client.? 172.21.15.137:0/4244508471' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:44:51.023 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:51 smithi059 ceph-mon[28424]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:44:51.322 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-23T08:44:51.322 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph mon dump -f json 2024-01-23T08:44:51.570 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-23T08:44:51.616 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-23T08:44:53.059 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.060+0000 7f30e6bae700 1 -- 172.21.15.137:0/100255429 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e00fb950 msgr2=0x7f30e00fbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:53.059 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.060+0000 7f30e6bae700 1 --2- 172.21.15.137:0/100255429 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e00fb950 0x7f30e00fbd60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f30d40048c0 tx=0x7f30d4009e30 comp rx=0 tx=0).stop 2024-01-23T08:44:53.059 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.060+0000 7f30e6bae700 1 -- 172.21.15.137:0/100255429 shutdown_connections 2024-01-23T08:44:53.059 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.060+0000 7f30e6bae700 1 --2- 172.21.15.137:0/100255429 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e00fb950 0x7f30e00fbd60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:53.059 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.060+0000 7f30e6bae700 1 -- 172.21.15.137:0/100255429 >> 172.21.15.137:0/100255429 conn(0x7f30e00f7030 msgr2=0x7f30e00f9440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:53.059 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.060+0000 7f30e6bae700 1 -- 172.21.15.137:0/100255429 shutdown_connections 2024-01-23T08:44:53.059 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.060+0000 7f30e6bae700 1 -- 172.21.15.137:0/100255429 wait complete. 2024-01-23T08:44:53.059 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.061+0000 7f30e6bae700 1 Processor -- start 2024-01-23T08:44:53.059 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.061+0000 7f30e6bae700 1 -- start start 2024-01-23T08:44:53.059 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.061+0000 7f30e6bae700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e00fb950 0x7f30e010f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:53.059 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.061+0000 7f30e6bae700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e00666a0 con 0x7f30e00fb950 2024-01-23T08:44:53.061 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.061+0000 7f30e494a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e00fb950 0x7f30e010f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:53.061 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.061+0000 7f30e494a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e00fb950 0x7f30e010f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.137:34420/0 (socket says 172.21.15.137:34420) 2024-01-23T08:44:53.061 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.061+0000 7f30e494a700 1 -- 172.21.15.137:0/2467451709 learned_addr learned my addr 172.21.15.137:0/2467451709 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-23T08:44:53.061 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.061+0000 7f30e494a700 1 -- 172.21.15.137:0/2467451709 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d4005040 con 0x7f30e00fb950 2024-01-23T08:44:53.062 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.062+0000 7f30e494a700 1 --2- 172.21.15.137:0/2467451709 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e00fb950 0x7f30e010f9c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f30d4009330 tx=0x7f30d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:53.062 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.062+0000 7f30dd7fa700 1 -- 172.21.15.137:0/2467451709 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30d40096b0 con 0x7f30e00fb950 2024-01-23T08:44:53.062 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.062+0000 7f30dd7fa700 1 -- 172.21.15.137:0/2467451709 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f30d4009810 con 0x7f30e00fb950 2024-01-23T08:44:53.062 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.062+0000 7f30e6bae700 1 -- 172.21.15.137:0/2467451709 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e00668a0 con 0x7f30e00fb950 2024-01-23T08:44:53.062 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.062+0000 7f30dd7fa700 1 -- 172.21.15.137:0/2467451709 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30d401c5b0 con 0x7f30e00fb950 2024-01-23T08:44:53.062 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.062+0000 7f30e6bae700 1 -- 172.21.15.137:0/2467451709 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e0110140 con 0x7f30e00fb950 2024-01-23T08:44:53.062 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.063+0000 7f30dd7fa700 1 -- 172.21.15.137:0/2467451709 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f30d401c870 con 0x7f30e00fb950 2024-01-23T08:44:53.062 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.063+0000 7f30e6bae700 1 -- 172.21.15.137:0/2467451709 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e01055e0 con 0x7f30e00fb950 2024-01-23T08:44:53.062 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.063+0000 7f30dd7fa700 1 --2- 172.21.15.137:0/2467451709 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f30d00366d0 0x7f30d0038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:53.062 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.063+0000 7f30dd7fa700 1 -- 172.21.15.137:0/2467451709 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f30d4049ae0 con 0x7f30e00fb950 2024-01-23T08:44:53.062 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.063+0000 7f30dffff700 1 --2- 172.21.15.137:0/2467451709 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f30d00366d0 0x7f30d0038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:53.062 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.064+0000 7f30dffff700 1 --2- 172.21.15.137:0/2467451709 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f30d00366d0 0x7f30d0038b80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f30cc009990 tx=0x7f30cc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:53.068 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.070+0000 7f30dd7fa700 1 -- 172.21.15.137:0/2467451709 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f30d4011dd0 con 0x7f30e00fb950 2024-01-23T08:44:53.108 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:53 smithi059 ceph-mon[28424]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:44:53.108 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:53 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:53.108 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:53 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:53.109 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:53 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:53.109 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:53 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:53.109 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:53 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:53.109 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:53 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:53.323 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.325+0000 7f30e6bae700 1 -- 172.21.15.137:0/2467451709 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30e004bd60 con 0x7f30e00fb950 2024-01-23T08:44:53.323 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.325+0000 7f30dd7fa700 1 -- 172.21.15.137:0/2467451709 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f30d4011370 con 0x7f30e00fb950 2024-01-23T08:44:53.324 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:44:53.324 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"3fc07a38-b9cb-11ee-95b1-87774f69a715","modified":"2024-01-23T08:43:01.494526Z","created":"2024-01-23T08:43:01.494526Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-23T08:44:53.325 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.327+0000 7f30e6bae700 1 -- 172.21.15.137:0/2467451709 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f30d00366d0 msgr2=0x7f30d0038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:53.325 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.327+0000 7f30e6bae700 1 --2- 172.21.15.137:0/2467451709 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f30d00366d0 0x7f30d0038b80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f30cc009990 tx=0x7f30cc006e40 comp rx=0 tx=0).stop 2024-01-23T08:44:53.325 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.327+0000 7f30e6bae700 1 -- 172.21.15.137:0/2467451709 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e00fb950 msgr2=0x7f30e010f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:53.325 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.327+0000 7f30e6bae700 1 --2- 172.21.15.137:0/2467451709 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e00fb950 0x7f30e010f9c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f30d4009330 tx=0x7f30d4009e30 comp rx=0 tx=0).stop 2024-01-23T08:44:53.326 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.327+0000 7f30e6bae700 1 -- 172.21.15.137:0/2467451709 shutdown_connections 2024-01-23T08:44:53.326 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.327+0000 7f30e6bae700 1 --2- 172.21.15.137:0/2467451709 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f30d00366d0 0x7f30d0038b80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:53.326 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.327+0000 7f30e6bae700 1 --2- 172.21.15.137:0/2467451709 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30e00fb950 0x7f30e010f9c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:53.326 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.327+0000 7f30e6bae700 1 -- 172.21.15.137:0/2467451709 >> 172.21.15.137:0/2467451709 conn(0x7f30e00f7030 msgr2=0x7f30e00f7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:53.326 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.328+0000 7f30e6bae700 1 -- 172.21.15.137:0/2467451709 shutdown_connections 2024-01-23T08:44:53.326 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:53.328+0000 7f30e6bae700 1 -- 172.21.15.137:0/2467451709 wait complete. 2024-01-23T08:44:53.326 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-01-23T08:44:54.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:54 smithi059 ceph-mon[28424]: Deploying daemon prometheus.smithi059 on smithi059 2024-01-23T08:44:54.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:54 smithi059 ceph-mon[28424]: from='client.? 172.21.15.137:0/2467451709' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:44:54.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:54 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:44:54.966 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-23T08:44:54.967 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph mon dump -f json 2024-01-23T08:44:55.213 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-23T08:44:55.259 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-23T08:44:55.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:55 smithi059 ceph-mon[28424]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:44:56.581 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.583+0000 7fb39104f700 1 -- 172.21.15.137:0/3699423442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c071da0 msgr2=0x7fb38c070210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:56.581 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.583+0000 7fb39104f700 1 --2- 172.21.15.137:0/3699423442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c071da0 0x7fb38c070210 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb37c0048c0 tx=0x7fb37c009e30 comp rx=0 tx=0).stop 2024-01-23T08:44:56.581 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.583+0000 7fb39104f700 1 -- 172.21.15.137:0/3699423442 shutdown_connections 2024-01-23T08:44:56.581 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.583+0000 7fb39104f700 1 --2- 172.21.15.137:0/3699423442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c071da0 0x7fb38c070210 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:56.581 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.583+0000 7fb39104f700 1 -- 172.21.15.137:0/3699423442 >> 172.21.15.137:0/3699423442 conn(0x7fb38c0f8130 msgr2=0x7fb38c0fa540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:56.581 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.583+0000 7fb39104f700 1 -- 172.21.15.137:0/3699423442 shutdown_connections 2024-01-23T08:44:56.581 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.583+0000 7fb39104f700 1 -- 172.21.15.137:0/3699423442 wait complete. 2024-01-23T08:44:56.582 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.583+0000 7fb39104f700 1 Processor -- start 2024-01-23T08:44:56.582 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.583+0000 7fb39104f700 1 -- start start 2024-01-23T08:44:56.582 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.583+0000 7fb39104f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c071da0 0x7fb38c113280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:56.583 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.583+0000 7fb39104f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38c1137c0 con 0x7fb38c071da0 2024-01-23T08:44:56.583 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.584+0000 7fb38ad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c071da0 0x7fb38c113280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:56.583 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.584+0000 7fb38ad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c071da0 0x7fb38c113280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.137:34452/0 (socket says 172.21.15.137:34452) 2024-01-23T08:44:56.583 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.584+0000 7fb38ad9d700 1 -- 172.21.15.137:0/1109412114 learned_addr learned my addr 172.21.15.137:0/1109412114 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-23T08:44:56.583 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.584+0000 7fb38ad9d700 1 -- 172.21.15.137:0/1109412114 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37c005040 con 0x7fb38c071da0 2024-01-23T08:44:56.583 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.584+0000 7fb38ad9d700 1 --2- 172.21.15.137:0/1109412114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c071da0 0x7fb38c113280 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb37c00b690 tx=0x7fb37c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:56.583 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.584+0000 7fb3737fe700 1 -- 172.21.15.137:0/1109412114 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb37c009650 con 0x7fb38c071da0 2024-01-23T08:44:56.583 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.584+0000 7fb39104f700 1 -- 172.21.15.137:0/1109412114 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb38c1139c0 con 0x7fb38c071da0 2024-01-23T08:44:56.583 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.584+0000 7fb3737fe700 1 -- 172.21.15.137:0/1109412114 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb37c0097b0 con 0x7fb38c071da0 2024-01-23T08:44:56.583 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.584+0000 7fb3737fe700 1 -- 172.21.15.137:0/1109412114 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb37c01c5b0 con 0x7fb38c071da0 2024-01-23T08:44:56.583 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.584+0000 7fb39104f700 1 -- 172.21.15.137:0/1109412114 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb38c114660 con 0x7fb38c071da0 2024-01-23T08:44:56.584 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.585+0000 7fb39104f700 1 -- 172.21.15.137:0/1109412114 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb38c05f450 con 0x7fb38c071da0 2024-01-23T08:44:56.584 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.585+0000 7fb3737fe700 1 -- 172.21.15.137:0/1109412114 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fb37c013420 con 0x7fb38c071da0 2024-01-23T08:44:56.584 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.586+0000 7fb3737fe700 1 --2- 172.21.15.137:0/1109412114 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fb3740366d0 0x7fb374038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:44:56.584 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.586+0000 7fb3737fe700 1 -- 172.21.15.137:0/1109412114 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb37c049910 con 0x7fb38c071da0 2024-01-23T08:44:56.585 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.586+0000 7fb38a59c700 1 --2- 172.21.15.137:0/1109412114 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fb3740366d0 0x7fb374038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:44:56.585 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.586+0000 7fb38a59c700 1 --2- 172.21.15.137:0/1109412114 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fb3740366d0 0x7fb374038b80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb380009990 tx=0x7fb380006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:44:56.589 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.592+0000 7fb3737fe700 1 -- 172.21.15.137:0/1109412114 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb37c013a60 con 0x7fb38c071da0 2024-01-23T08:44:56.847 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.849+0000 7fb39104f700 1 -- 172.21.15.137:0/1109412114 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb38c1151b0 con 0x7fb38c071da0 2024-01-23T08:44:56.847 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.849+0000 7fb3737fe700 1 -- 172.21.15.137:0/1109412114 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fb37c011370 con 0x7fb38c071da0 2024-01-23T08:44:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:44:56.848 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"3fc07a38-b9cb-11ee-95b1-87774f69a715","modified":"2024-01-23T08:43:01.494526Z","created":"2024-01-23T08:43:01.494526Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-23T08:44:56.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.851+0000 7fb39104f700 1 -- 172.21.15.137:0/1109412114 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fb3740366d0 msgr2=0x7fb374038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:56.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.851+0000 7fb39104f700 1 --2- 172.21.15.137:0/1109412114 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fb3740366d0 0x7fb374038b80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb380009990 tx=0x7fb380006e40 comp rx=0 tx=0).stop 2024-01-23T08:44:56.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.851+0000 7fb39104f700 1 -- 172.21.15.137:0/1109412114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c071da0 msgr2=0x7fb38c113280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:44:56.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.851+0000 7fb39104f700 1 --2- 172.21.15.137:0/1109412114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c071da0 0x7fb38c113280 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb37c00b690 tx=0x7fb37c009e30 comp rx=0 tx=0).stop 2024-01-23T08:44:56.850 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.851+0000 7fb39104f700 1 -- 172.21.15.137:0/1109412114 shutdown_connections 2024-01-23T08:44:56.850 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.851+0000 7fb39104f700 1 --2- 172.21.15.137:0/1109412114 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fb3740366d0 0x7fb374038b80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:56.850 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.851+0000 7fb39104f700 1 --2- 172.21.15.137:0/1109412114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c071da0 0x7fb38c113280 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:44:56.850 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.851+0000 7fb39104f700 1 -- 172.21.15.137:0/1109412114 >> 172.21.15.137:0/1109412114 conn(0x7fb38c0f8130 msgr2=0x7fb38c0f8ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:44:56.850 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.851+0000 7fb39104f700 1 -- 172.21.15.137:0/1109412114 shutdown_connections 2024-01-23T08:44:56.850 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:44:56.851+0000 7fb39104f700 1 -- 172.21.15.137:0/1109412114 wait complete. 2024-01-23T08:44:56.850 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-01-23T08:44:57.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:57 smithi059 ceph-mon[28424]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:44:57.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:57 smithi059 ceph-mon[28424]: from='client.? 172.21.15.137:0/1109412114' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:44:58.394 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-23T08:44:58.394 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph mon dump -f json 2024-01-23T08:44:58.642 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-23T08:44:58.689 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-23T08:44:59.271 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:44:59 smithi059 ceph-mon[28424]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:00.141 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.142+0000 7f4ab739c700 1 -- 172.21.15.137:0/2056205453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ab00fb960 msgr2=0x7f4ab00fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:00.142 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.142+0000 7f4ab739c700 1 --2- 172.21.15.137:0/2056205453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ab00fb960 0x7f4ab00fbd70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4aac0048c0 tx=0x7f4aac009e30 comp rx=0 tx=0).stop 2024-01-23T08:45:00.142 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.142+0000 7f4ab739c700 1 -- 172.21.15.137:0/2056205453 shutdown_connections 2024-01-23T08:45:00.142 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.142+0000 7f4ab739c700 1 --2- 172.21.15.137:0/2056205453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ab00fb960 0x7f4ab00fbd70 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:00.142 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.142+0000 7f4ab739c700 1 -- 172.21.15.137:0/2056205453 >> 172.21.15.137:0/2056205453 conn(0x7f4ab00f7020 msgr2=0x7f4ab00f9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:45:00.142 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.143+0000 7f4ab739c700 1 -- 172.21.15.137:0/2056205453 shutdown_connections 2024-01-23T08:45:00.142 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.143+0000 7f4ab739c700 1 -- 172.21.15.137:0/2056205453 wait complete. 2024-01-23T08:45:00.142 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.143+0000 7f4ab739c700 1 Processor -- start 2024-01-23T08:45:00.142 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.143+0000 7f4ab739c700 1 -- start start 2024-01-23T08:45:00.142 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.143+0000 7f4ab739c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ab00fb960 0x7f4ab010f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:00.142 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.143+0000 7f4ab739c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab0065bf0 con 0x7f4ab00fb960 2024-01-23T08:45:00.142 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.144+0000 7f4ab5138700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ab00fb960 0x7f4ab010f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:45:00.142 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.144+0000 7f4ab5138700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ab00fb960 0x7f4ab010f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.137:34478/0 (socket says 172.21.15.137:34478) 2024-01-23T08:45:00.143 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.144+0000 7f4ab5138700 1 -- 172.21.15.137:0/648622975 learned_addr learned my addr 172.21.15.137:0/648622975 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-23T08:45:00.143 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.144+0000 7f4ab5138700 1 -- 172.21.15.137:0/648622975 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac005040 con 0x7f4ab00fb960 2024-01-23T08:45:00.143 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.144+0000 7f4ab5138700 1 --2- 172.21.15.137:0/648622975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ab00fb960 0x7f4ab010f9a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4aac00b690 tx=0x7f4aac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:45:00.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.144+0000 7f4aa5ffb700 1 -- 172.21.15.137:0/648622975 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4aac009650 con 0x7f4ab00fb960 2024-01-23T08:45:00.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.144+0000 7f4aa5ffb700 1 -- 172.21.15.137:0/648622975 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4aac0097b0 con 0x7f4ab00fb960 2024-01-23T08:45:00.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.144+0000 7f4ab739c700 1 -- 172.21.15.137:0/648622975 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab0065df0 con 0x7f4ab00fb960 2024-01-23T08:45:00.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.144+0000 7f4aa5ffb700 1 -- 172.21.15.137:0/648622975 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4aac01c5d0 con 0x7f4ab00fb960 2024-01-23T08:45:00.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.144+0000 7f4ab739c700 1 -- 172.21.15.137:0/648622975 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ab0110120 con 0x7f4ab00fb960 2024-01-23T08:45:00.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.145+0000 7f4ab739c700 1 -- 172.21.15.137:0/648622975 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ab01055f0 con 0x7f4ab00fb960 2024-01-23T08:45:00.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.145+0000 7f4aa5ffb700 1 -- 172.21.15.137:0/648622975 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f4aac0139d0 con 0x7f4ab00fb960 2024-01-23T08:45:00.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.146+0000 7f4aa5ffb700 1 --2- 172.21.15.137:0/648622975 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f4a9c0366d0 0x7f4a9c038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:00.145 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.146+0000 7f4aa5ffb700 1 -- 172.21.15.137:0/648622975 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4aac049a00 con 0x7f4ab00fb960 2024-01-23T08:45:00.145 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.146+0000 7f4ab4937700 1 --2- 172.21.15.137:0/648622975 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f4a9c0366d0 0x7f4a9c038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:45:00.145 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.146+0000 7f4ab4937700 1 --2- 172.21.15.137:0/648622975 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f4a9c0366d0 0x7f4a9c038b80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0009990 tx=0x7f4aa0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:45:00.148 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.151+0000 7f4aa5ffb700 1 -- 172.21.15.137:0/648622975 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f4aac025400 con 0x7f4ab00fb960 2024-01-23T08:45:00.403 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.405+0000 7f4ab739c700 1 -- 172.21.15.137:0/648622975 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ab004bd60 con 0x7f4ab00fb960 2024-01-23T08:45:00.403 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.406+0000 7f4aa5ffb700 1 -- 172.21.15.137:0/648622975 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f4aac011b60 con 0x7f4ab00fb960 2024-01-23T08:45:00.403 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:45:00.404 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"3fc07a38-b9cb-11ee-95b1-87774f69a715","modified":"2024-01-23T08:43:01.494526Z","created":"2024-01-23T08:43:01.494526Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-23T08:45:00.406 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.408+0000 7f4ab739c700 1 -- 172.21.15.137:0/648622975 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f4a9c0366d0 msgr2=0x7f4a9c038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:00.406 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.408+0000 7f4ab739c700 1 --2- 172.21.15.137:0/648622975 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f4a9c0366d0 0x7f4a9c038b80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0009990 tx=0x7f4aa0006e40 comp rx=0 tx=0).stop 2024-01-23T08:45:00.406 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.408+0000 7f4ab739c700 1 -- 172.21.15.137:0/648622975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ab00fb960 msgr2=0x7f4ab010f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:00.406 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.408+0000 7f4ab739c700 1 --2- 172.21.15.137:0/648622975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ab00fb960 0x7f4ab010f9a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4aac00b690 tx=0x7f4aac009e30 comp rx=0 tx=0).stop 2024-01-23T08:45:00.406 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.408+0000 7f4ab739c700 1 -- 172.21.15.137:0/648622975 shutdown_connections 2024-01-23T08:45:00.406 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.408+0000 7f4ab739c700 1 --2- 172.21.15.137:0/648622975 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f4a9c0366d0 0x7f4a9c038b80 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:00.406 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.408+0000 7f4ab739c700 1 --2- 172.21.15.137:0/648622975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ab00fb960 0x7f4ab010f9a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:00.406 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.408+0000 7f4ab739c700 1 -- 172.21.15.137:0/648622975 >> 172.21.15.137:0/648622975 conn(0x7f4ab00f7020 msgr2=0x7f4ab00f7e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:45:00.406 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.408+0000 7f4ab739c700 1 -- 172.21.15.137:0/648622975 shutdown_connections 2024-01-23T08:45:00.407 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:00.408+0000 7f4ab739c700 1 -- 172.21.15.137:0/648622975 wait complete. 2024-01-23T08:45:00.407 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-01-23T08:45:01.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:01 smithi059 ceph-mon[28424]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:01.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:01 smithi059 ceph-mon[28424]: from='client.? 172.21.15.137:0/648622975' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:01.956 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-23T08:45:01.957 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph mon dump -f json 2024-01-23T08:45:02.204 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-23T08:45:02.250 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-23T08:45:03.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:03 smithi059 ceph-mon[28424]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:03.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:03 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:03.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:03 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:03.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:03 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:03.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:03 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-01-23T08:45:03.742 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.726+0000 7fbf4fb29700 1 -- 172.21.15.137:0/667564890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf480666d0 msgr2=0x7fbf480ff660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:03.742 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.726+0000 7fbf4fb29700 1 --2- 172.21.15.137:0/667564890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf480666d0 0x7fbf480ff660 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009c60 tx=0x7fbf40004d10 comp rx=0 tx=0).stop 2024-01-23T08:45:03.742 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.727+0000 7fbf4fb29700 1 -- 172.21.15.137:0/667564890 shutdown_connections 2024-01-23T08:45:03.742 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.727+0000 7fbf4fb29700 1 --2- 172.21.15.137:0/667564890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf480666d0 0x7fbf480ff660 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:03.742 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.727+0000 7fbf4fb29700 1 -- 172.21.15.137:0/667564890 >> 172.21.15.137:0/667564890 conn(0x7fbf480f7030 msgr2=0x7fbf480f9440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:45:03.742 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.727+0000 7fbf4fb29700 1 -- 172.21.15.137:0/667564890 shutdown_connections 2024-01-23T08:45:03.742 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.727+0000 7fbf4fb29700 1 -- 172.21.15.137:0/667564890 wait complete. 2024-01-23T08:45:03.742 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.727+0000 7fbf4fb29700 1 Processor -- start 2024-01-23T08:45:03.742 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.727+0000 7fbf4fb29700 1 -- start start 2024-01-23T08:45:03.742 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.727+0000 7fbf4fb29700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf480666d0 0x7fbf4810cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:03.742 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.727+0000 7fbf4fb29700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4810d2d0 con 0x7fbf480666d0 2024-01-23T08:45:03.742 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.728+0000 7fbf4d8c5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf480666d0 0x7fbf4810cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:45:03.742 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.728+0000 7fbf4d8c5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf480666d0 0x7fbf4810cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.137:39548/0 (socket says 172.21.15.137:39548) 2024-01-23T08:45:03.743 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.728+0000 7fbf4d8c5700 1 -- 172.21.15.137:0/560299928 learned_addr learned my addr 172.21.15.137:0/560299928 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-23T08:45:03.743 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.728+0000 7fbf4d8c5700 1 -- 172.21.15.137:0/560299928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf40009940 con 0x7fbf480666d0 2024-01-23T08:45:03.743 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.728+0000 7fbf4d8c5700 1 --2- 172.21.15.137:0/560299928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf480666d0 0x7fbf4810cd90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbf40007b30 tx=0x7fbf40004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:45:03.743 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.728+0000 7fbf467fc700 1 -- 172.21.15.137:0/560299928 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf4000bec0 con 0x7fbf480666d0 2024-01-23T08:45:03.743 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.728+0000 7fbf4fb29700 1 -- 172.21.15.137:0/560299928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf4810d4d0 con 0x7fbf480666d0 2024-01-23T08:45:03.743 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.728+0000 7fbf467fc700 1 -- 172.21.15.137:0/560299928 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf4001d8f0 con 0x7fbf480666d0 2024-01-23T08:45:03.743 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.728+0000 7fbf467fc700 1 -- 172.21.15.137:0/560299928 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf400128e0 con 0x7fbf480666d0 2024-01-23T08:45:03.743 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.728+0000 7fbf4fb29700 1 -- 172.21.15.137:0/560299928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf4810e0c0 con 0x7fbf480666d0 2024-01-23T08:45:03.743 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.729+0000 7fbf4fb29700 1 -- 172.21.15.137:0/560299928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf4805f450 con 0x7fbf480666d0 2024-01-23T08:45:03.743 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.729+0000 7fbf467fc700 1 -- 172.21.15.137:0/560299928 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 14) v1 ==== 43351+0+0 (secure 0 0 0) 0x7fbf40019040 con 0x7fbf480666d0 2024-01-23T08:45:03.743 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.729+0000 7fbf467fc700 1 --2- 172.21.15.137:0/560299928 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fbf30036720 0x7fbf30038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:03.743 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.729+0000 7fbf467fc700 1 -- 172.21.15.137:0/560299928 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fbf4004a250 con 0x7fbf480666d0 2024-01-23T08:45:03.743 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.729+0000 7fbf4d0c4700 1 -- 172.21.15.137:0/560299928 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fbf30036720 msgr2=0x7fbf30038bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3614124404 2024-01-23T08:45:03.743 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.729+0000 7fbf4d0c4700 1 --2- 172.21.15.137:0/560299928 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fbf30036720 0x7fbf30038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-23T08:45:03.743 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.737+0000 7fbf467fc700 1 -- 172.21.15.137:0/560299928 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fbf4002eb40 con 0x7fbf480666d0 2024-01-23T08:45:03.928 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.930+0000 7fbf4d0c4700 1 -- 172.21.15.137:0/560299928 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fbf30036720 msgr2=0x7fbf30038bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3614124404 2024-01-23T08:45:03.928 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.930+0000 7fbf4d0c4700 1 --2- 172.21.15.137:0/560299928 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fbf30036720 0x7fbf30038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-23T08:45:03.988 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.990+0000 7fbf4fb29700 1 -- 172.21.15.137:0/560299928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbf4804bd60 con 0x7fbf480666d0 2024-01-23T08:45:03.989 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.991+0000 7fbf467fc700 1 -- 172.21.15.137:0/560299928 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fbf40021020 con 0x7fbf480666d0 2024-01-23T08:45:03.989 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:45:03.989 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"3fc07a38-b9cb-11ee-95b1-87774f69a715","modified":"2024-01-23T08:43:01.494526Z","created":"2024-01-23T08:43:01.494526Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-23T08:45:03.991 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.993+0000 7fbf4fb29700 1 -- 172.21.15.137:0/560299928 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fbf30036720 msgr2=0x7fbf30038bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-23T08:45:03.991 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.993+0000 7fbf4fb29700 1 --2- 172.21.15.137:0/560299928 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fbf30036720 0x7fbf30038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:03.991 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.993+0000 7fbf4fb29700 1 -- 172.21.15.137:0/560299928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf480666d0 msgr2=0x7fbf4810cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:03.991 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.993+0000 7fbf4fb29700 1 --2- 172.21.15.137:0/560299928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf480666d0 0x7fbf4810cd90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbf40007b30 tx=0x7fbf40004d10 comp rx=0 tx=0).stop 2024-01-23T08:45:03.991 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.993+0000 7fbf4fb29700 1 -- 172.21.15.137:0/560299928 shutdown_connections 2024-01-23T08:45:03.992 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.993+0000 7fbf4fb29700 1 --2- 172.21.15.137:0/560299928 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7fbf30036720 0x7fbf30038bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:03.992 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.993+0000 7fbf4fb29700 1 --2- 172.21.15.137:0/560299928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf480666d0 0x7fbf4810cd90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:03.992 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.993+0000 7fbf4fb29700 1 -- 172.21.15.137:0/560299928 >> 172.21.15.137:0/560299928 conn(0x7fbf480f7030 msgr2=0x7fbf480f7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:45:03.992 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.993+0000 7fbf4fb29700 1 -- 172.21.15.137:0/560299928 shutdown_connections 2024-01-23T08:45:03.992 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:03.993+0000 7fbf4fb29700 1 -- 172.21.15.137:0/560299928 wait complete. 2024-01-23T08:45:03.992 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-01-23T08:45:04.866 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:04 smithi059 ceph-mon[28424]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-01-23T08:45:04.866 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:04 smithi059 ceph-mon[28424]: mgrmap e14: smithi059.bzqirf(active, since 59s) 2024-01-23T08:45:04.867 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:04 smithi059 ceph-mon[28424]: from='client.? 172.21.15.137:0/560299928' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:05.569 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-23T08:45:05.570 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph mon dump -f json 2024-01-23T08:45:05.816 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-23T08:45:05.862 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-23T08:45:07.240 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.241+0000 7f1ea6563700 1 -- 172.21.15.137:0/3141372733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ea00fb950 msgr2=0x7f1ea00fbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:07.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.241+0000 7f1ea6563700 1 --2- 172.21.15.137:0/3141372733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ea00fb950 0x7f1ea00fbd60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1e900048c0 tx=0x7f1e90009e30 comp rx=0 tx=0).stop 2024-01-23T08:45:07.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.241+0000 7f1ea6563700 1 -- 172.21.15.137:0/3141372733 shutdown_connections 2024-01-23T08:45:07.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.241+0000 7f1ea6563700 1 --2- 172.21.15.137:0/3141372733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ea00fb950 0x7f1ea00fbd60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:07.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.241+0000 7f1ea6563700 1 -- 172.21.15.137:0/3141372733 >> 172.21.15.137:0/3141372733 conn(0x7f1ea00f7030 msgr2=0x7f1ea00f9440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:45:07.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.241+0000 7f1ea6563700 1 -- 172.21.15.137:0/3141372733 shutdown_connections 2024-01-23T08:45:07.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.241+0000 7f1ea6563700 1 -- 172.21.15.137:0/3141372733 wait complete. 2024-01-23T08:45:07.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.241+0000 7f1ea6563700 1 Processor -- start 2024-01-23T08:45:07.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.241+0000 7f1ea6563700 1 -- start start 2024-01-23T08:45:07.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.241+0000 7f1ea6563700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ea00fb950 0x7f1ea010f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:07.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.242+0000 7f1ea6563700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea0066700 con 0x7f1ea00fb950 2024-01-23T08:45:07.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.242+0000 7f1e9ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ea00fb950 0x7f1ea010f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:45:07.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.242+0000 7f1e9ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ea00fb950 0x7f1ea010f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.137:39564/0 (socket says 172.21.15.137:39564) 2024-01-23T08:45:07.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.242+0000 7f1e9ffff700 1 -- 172.21.15.137:0/1544206647 learned_addr learned my addr 172.21.15.137:0/1544206647 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-23T08:45:07.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.242+0000 7f1e9ffff700 1 -- 172.21.15.137:0/1544206647 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e90005040 con 0x7f1ea00fb950 2024-01-23T08:45:07.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.242+0000 7f1e9ffff700 1 --2- 172.21.15.137:0/1544206647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ea00fb950 0x7f1ea010f960 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1e90009330 tx=0x7f1e90009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:45:07.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.242+0000 7f1e9cff9700 1 -- 172.21.15.137:0/1544206647 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e900096b0 con 0x7f1ea00fb950 2024-01-23T08:45:07.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.242+0000 7f1e9cff9700 1 -- 172.21.15.137:0/1544206647 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e90009810 con 0x7f1ea00fb950 2024-01-23T08:45:07.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.242+0000 7f1ea6563700 1 -- 172.21.15.137:0/1544206647 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea0066900 con 0x7f1ea00fb950 2024-01-23T08:45:07.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.242+0000 7f1e9cff9700 1 -- 172.21.15.137:0/1544206647 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e9001c5b0 con 0x7f1ea00fb950 2024-01-23T08:45:07.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.242+0000 7f1ea6563700 1 -- 172.21.15.137:0/1544206647 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea01100e0 con 0x7f1ea00fb950 2024-01-23T08:45:07.243 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.243+0000 7f1e9cff9700 1 -- 172.21.15.137:0/1544206647 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 14) v1 ==== 43351+0+0 (secure 0 0 0) 0x7f1e9001c870 con 0x7f1ea00fb950 2024-01-23T08:45:07.243 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.243+0000 7f1e9cff9700 1 --2- 172.21.15.137:0/1544206647 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f1e88036720 0x7f1e88038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:07.243 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.243+0000 7f1e9cff9700 1 -- 172.21.15.137:0/1544206647 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1e90049b20 con 0x7f1ea00fb950 2024-01-23T08:45:07.243 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.243+0000 7f1e9f7fe700 1 -- 172.21.15.137:0/1544206647 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f1e88036720 msgr2=0x7f1e88038bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3614124404 2024-01-23T08:45:07.243 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.243+0000 7f1ea6563700 1 -- 172.21.15.137:0/1544206647 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ea01055e0 con 0x7f1ea00fb950 2024-01-23T08:45:07.243 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.243+0000 7f1e9f7fe700 1 --2- 172.21.15.137:0/1544206647 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f1e88036720 0x7f1e88038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-23T08:45:07.250 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.252+0000 7f1e9cff9700 1 -- 172.21.15.137:0/1544206647 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1e90011bd0 con 0x7f1ea00fb950 2024-01-23T08:45:07.442 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.444+0000 7f1e9f7fe700 1 -- 172.21.15.137:0/1544206647 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f1e88036720 msgr2=0x7f1e88038bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3614124404 2024-01-23T08:45:07.442 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.444+0000 7f1e9f7fe700 1 --2- 172.21.15.137:0/1544206647 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f1e88036720 0x7f1e88038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-23T08:45:07.506 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.508+0000 7f1ea6563700 1 -- 172.21.15.137:0/1544206647 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1ea004bd60 con 0x7f1ea00fb950 2024-01-23T08:45:07.507 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.509+0000 7f1e9cff9700 1 -- 172.21.15.137:0/1544206647 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f1e90034090 con 0x7f1ea00fb950 2024-01-23T08:45:07.507 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:45:07.507 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"3fc07a38-b9cb-11ee-95b1-87774f69a715","modified":"2024-01-23T08:43:01.494526Z","created":"2024-01-23T08:43:01.494526Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-23T08:45:07.509 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.511+0000 7f1ea6563700 1 -- 172.21.15.137:0/1544206647 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f1e88036720 msgr2=0x7f1e88038bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-23T08:45:07.509 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.511+0000 7f1ea6563700 1 --2- 172.21.15.137:0/1544206647 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f1e88036720 0x7f1e88038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:07.509 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.511+0000 7f1ea6563700 1 -- 172.21.15.137:0/1544206647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ea00fb950 msgr2=0x7f1ea010f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:07.509 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.511+0000 7f1ea6563700 1 --2- 172.21.15.137:0/1544206647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ea00fb950 0x7f1ea010f960 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1e90009330 tx=0x7f1e90009e30 comp rx=0 tx=0).stop 2024-01-23T08:45:07.509 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.511+0000 7f1ea6563700 1 -- 172.21.15.137:0/1544206647 shutdown_connections 2024-01-23T08:45:07.509 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.511+0000 7f1ea6563700 1 --2- 172.21.15.137:0/1544206647 >> [v2:172.21.15.59:6800/3614124404,v1:172.21.15.59:6801/3614124404] conn(0x7f1e88036720 0x7f1e88038bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:07.509 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.511+0000 7f1ea6563700 1 --2- 172.21.15.137:0/1544206647 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ea00fb950 0x7f1ea010f960 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:07.510 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.511+0000 7f1ea6563700 1 -- 172.21.15.137:0/1544206647 >> 172.21.15.137:0/1544206647 conn(0x7f1ea00f7030 msgr2=0x7f1ea00f7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:45:07.510 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.511+0000 7f1ea6563700 1 -- 172.21.15.137:0/1544206647 shutdown_connections 2024-01-23T08:45:07.510 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:07.511+0000 7f1ea6563700 1 -- 172.21.15.137:0/1544206647 wait complete. 2024-01-23T08:45:07.510 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-01-23T08:45:07.909 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:07 smithi059 ceph-mon[28424]: from='client.? 172.21.15.137:0/1544206647' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:09.081 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-23T08:45:09.081 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph mon dump -f json 2024-01-23T08:45:09.329 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-23T08:45:09.375 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-23T08:45:10.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.850+0000 7feb6b6d4700 1 -- 172.21.15.137:0/83484117 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb640fd5a0 msgr2=0x7feb640ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:10.850 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.850+0000 7feb6b6d4700 1 --2- 172.21.15.137:0/83484117 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb640fd5a0 0x7feb640ff980 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7feb600048c0 tx=0x7feb60009e30 comp rx=0 tx=0).stop 2024-01-23T08:45:10.850 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.850+0000 7feb6b6d4700 1 -- 172.21.15.137:0/83484117 shutdown_connections 2024-01-23T08:45:10.850 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.850+0000 7feb6b6d4700 1 --2- 172.21.15.137:0/83484117 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb640fd5a0 0x7feb640ff980 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:10.850 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.850+0000 7feb6b6d4700 1 -- 172.21.15.137:0/83484117 >> 172.21.15.137:0/83484117 conn(0x7feb640f7020 msgr2=0x7feb640f9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:45:10.850 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.850+0000 7feb6b6d4700 1 -- 172.21.15.137:0/83484117 shutdown_connections 2024-01-23T08:45:10.850 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.850+0000 7feb6b6d4700 1 -- 172.21.15.137:0/83484117 wait complete. 2024-01-23T08:45:10.850 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.850+0000 7feb6b6d4700 1 Processor -- start 2024-01-23T08:45:10.850 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.850+0000 7feb6b6d4700 1 -- start start 2024-01-23T08:45:10.850 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.851+0000 7feb6b6d4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb640fd5a0 0x7feb6410eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:10.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.851+0000 7feb6b6d4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6410f410 con 0x7feb640fd5a0 2024-01-23T08:45:10.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.851+0000 7feb69470700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb640fd5a0 0x7feb6410eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:45:10.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.851+0000 7feb69470700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb640fd5a0 0x7feb6410eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.137:39574/0 (socket says 172.21.15.137:39574) 2024-01-23T08:45:10.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.851+0000 7feb69470700 1 -- 172.21.15.137:0/742870957 learned_addr learned my addr 172.21.15.137:0/742870957 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-23T08:45:10.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.851+0000 7feb69470700 1 -- 172.21.15.137:0/742870957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb60005040 con 0x7feb640fd5a0 2024-01-23T08:45:10.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.851+0000 7feb69470700 1 --2- 172.21.15.137:0/742870957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb640fd5a0 0x7feb6410eed0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7feb6000b690 tx=0x7feb60009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:45:10.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.851+0000 7feb59ffb700 1 -- 172.21.15.137:0/742870957 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb60009650 con 0x7feb640fd5a0 2024-01-23T08:45:10.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.851+0000 7feb6b6d4700 1 -- 172.21.15.137:0/742870957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb6410f610 con 0x7feb640fd5a0 2024-01-23T08:45:10.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.851+0000 7feb59ffb700 1 -- 172.21.15.137:0/742870957 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb600097b0 con 0x7feb640fd5a0 2024-01-23T08:45:10.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.851+0000 7feb59ffb700 1 -- 172.21.15.137:0/742870957 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb6001c5b0 con 0x7feb640fd5a0 2024-01-23T08:45:10.852 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.852+0000 7feb6b6d4700 1 -- 172.21.15.137:0/742870957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb64110170 con 0x7feb640fd5a0 2024-01-23T08:45:10.852 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.853+0000 7feb59ffb700 1 -- 172.21.15.137:0/742870957 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 15) v1 ==== 43075+0+0 (secure 0 0 0) 0x7feb6001c870 con 0x7feb640fd5a0 2024-01-23T08:45:10.852 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.853+0000 7feb6b6d4700 1 -- 172.21.15.137:0/742870957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb6405f450 con 0x7feb640fd5a0 2024-01-23T08:45:10.852 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.853+0000 7feb59ffb700 1 -- 172.21.15.137:0/742870957 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7feb60049600 con 0x7feb640fd5a0 2024-01-23T08:45:10.856 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:10.858+0000 7feb59ffb700 1 -- 172.21.15.137:0/742870957 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7feb60023300 con 0x7feb640fd5a0 2024-01-23T08:45:11.129 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:11.130+0000 7feb6b6d4700 1 -- 172.21.15.137:0/742870957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb6404bd60 con 0x7feb640fd5a0 2024-01-23T08:45:11.129 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:11.131+0000 7feb59ffb700 1 -- 172.21.15.137:0/742870957 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7feb60011b60 con 0x7feb640fd5a0 2024-01-23T08:45:11.129 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:45:11.129 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"3fc07a38-b9cb-11ee-95b1-87774f69a715","modified":"2024-01-23T08:43:01.494526Z","created":"2024-01-23T08:43:01.494526Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-23T08:45:11.131 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:11.133+0000 7feb6b6d4700 1 -- 172.21.15.137:0/742870957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb640fd5a0 msgr2=0x7feb6410eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:11.131 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:11.133+0000 7feb6b6d4700 1 --2- 172.21.15.137:0/742870957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb640fd5a0 0x7feb6410eed0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7feb6000b690 tx=0x7feb60009e30 comp rx=0 tx=0).stop 2024-01-23T08:45:11.131 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:11.133+0000 7feb6b6d4700 1 -- 172.21.15.137:0/742870957 shutdown_connections 2024-01-23T08:45:11.131 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:11.133+0000 7feb6b6d4700 1 --2- 172.21.15.137:0/742870957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb640fd5a0 0x7feb6410eed0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:11.131 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:11.133+0000 7feb6b6d4700 1 -- 172.21.15.137:0/742870957 >> 172.21.15.137:0/742870957 conn(0x7feb640f7020 msgr2=0x7feb640f9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:45:11.131 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:11.133+0000 7feb6b6d4700 1 -- 172.21.15.137:0/742870957 shutdown_connections 2024-01-23T08:45:11.131 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:11.133+0000 7feb6b6d4700 1 -- 172.21.15.137:0/742870957 wait complete. 2024-01-23T08:45:11.132 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-01-23T08:45:11.245 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:10 smithi059 ceph-mon[28424]: Active manager daemon smithi059.bzqirf restarted 2024-01-23T08:45:11.245 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:10 smithi059 ceph-mon[28424]: Activating manager daemon smithi059.bzqirf 2024-01-23T08:45:11.245 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:10 smithi059 ceph-mon[28424]: osdmap e5: 0 total, 0 up, 0 in 2024-01-23T08:45:11.245 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:10 smithi059 ceph-mon[28424]: mgrmap e15: smithi059.bzqirf(active, starting, since 0.00306046s) 2024-01-23T08:45:11.245 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:10 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-01-23T08:45:11.245 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:10 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mgr metadata", "who": "smithi059.bzqirf", "id": "smithi059.bzqirf"}]: dispatch 2024-01-23T08:45:11.245 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:10 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-23T08:45:11.245 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:10 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-23T08:45:11.246 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:10 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-23T08:45:11.246 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:10 smithi059 ceph-mon[28424]: Manager daemon smithi059.bzqirf is now available 2024-01-23T08:45:11.246 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:10 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:45:11.246 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:10 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:45:11.246 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:10 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:11.246 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:10 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:45:11.246 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:10 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.bzqirf/mirror_snapshot_schedule"}]: dispatch 2024-01-23T08:45:11.246 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:10 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.bzqirf/trash_purge_schedule"}]: dispatch 2024-01-23T08:45:11.246 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:10 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:12.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:11 smithi059 ceph-mon[28424]: from='client.? 172.21.15.137:0/742870957' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:12.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:11 smithi059 ceph-mon[28424]: mgrmap e16: smithi059.bzqirf(active, since 1.00493s) 2024-01-23T08:45:12.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:11 smithi059 ceph-mon[28424]: [23/Jan/2024:08:45:11] ENGINE Bus STARTING 2024-01-23T08:45:12.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:11 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:12.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:11 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:12.743 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-23T08:45:12.744 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph mon dump -f json 2024-01-23T08:45:13.611 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-23T08:45:13.658 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-23T08:45:13.843 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:13 smithi059 ceph-mon[28424]: [23/Jan/2024:08:45:11] ENGINE Serving on https://172.21.15.59:7150 2024-01-23T08:45:13.843 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:13 smithi059 ceph-mon[28424]: [23/Jan/2024:08:45:11] ENGINE Bus STARTED 2024-01-23T08:45:13.843 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:13 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:13.843 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:13 smithi059 ceph-mon[28424]: mgrmap e17: smithi059.bzqirf(active, since 2s) 2024-01-23T08:45:15.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:15 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:15.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:15 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:15.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:15 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:15.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:15 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:15.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:15 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd/host:smithi059", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:45:16.074 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.076+0000 7f6ad1705700 1 -- 172.21.15.137:0/780151259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6acc0fd5a0 msgr2=0x7f6acc0ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:16.075 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.076+0000 7f6ad1705700 1 --2- 172.21.15.137:0/780151259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6acc0fd5a0 0x7f6acc0ff980 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6abc0048c0 tx=0x7f6abc009e30 comp rx=0 tx=0).stop 2024-01-23T08:45:16.075 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.076+0000 7f6ad1705700 1 -- 172.21.15.137:0/780151259 shutdown_connections 2024-01-23T08:45:16.075 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.076+0000 7f6ad1705700 1 --2- 172.21.15.137:0/780151259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6acc0fd5a0 0x7f6acc0ff980 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:16.075 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.076+0000 7f6ad1705700 1 -- 172.21.15.137:0/780151259 >> 172.21.15.137:0/780151259 conn(0x7f6acc0f7020 msgr2=0x7f6acc0f9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:45:16.075 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.076+0000 7f6ad1705700 1 -- 172.21.15.137:0/780151259 shutdown_connections 2024-01-23T08:45:16.075 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.076+0000 7f6ad1705700 1 -- 172.21.15.137:0/780151259 wait complete. 2024-01-23T08:45:16.075 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.077+0000 7f6ad1705700 1 Processor -- start 2024-01-23T08:45:16.075 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.077+0000 7f6ad1705700 1 -- start start 2024-01-23T08:45:16.075 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.077+0000 7f6ad1705700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6acc0fd5a0 0x7f6acc0fd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:16.075 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.077+0000 7f6ad1705700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc0fb4b0 con 0x7f6acc0fd5a0 2024-01-23T08:45:16.076 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.077+0000 7f6acaffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6acc0fd5a0 0x7f6acc0fd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:45:16.076 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.077+0000 7f6acaffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6acc0fd5a0 0x7f6acc0fd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.137:47416/0 (socket says 172.21.15.137:47416) 2024-01-23T08:45:16.077 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.077+0000 7f6acaffd700 1 -- 172.21.15.137:0/4045695746 learned_addr learned my addr 172.21.15.137:0/4045695746 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-23T08:45:16.077 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.077+0000 7f6acaffd700 1 -- 172.21.15.137:0/4045695746 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc005040 con 0x7f6acc0fd5a0 2024-01-23T08:45:16.077 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.078+0000 7f6acaffd700 1 --2- 172.21.15.137:0/4045695746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6acc0fd5a0 0x7f6acc0fd030 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6abc009040 tx=0x7f6abc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:45:16.077 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.078+0000 7f6ab37fe700 1 -- 172.21.15.137:0/4045695746 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6abc009620 con 0x7f6acc0fd5a0 2024-01-23T08:45:16.077 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.078+0000 7f6ab37fe700 1 -- 172.21.15.137:0/4045695746 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6abc009780 con 0x7f6acc0fd5a0 2024-01-23T08:45:16.077 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.078+0000 7f6ab37fe700 1 -- 172.21.15.137:0/4045695746 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6abc01c5b0 con 0x7f6acc0fd5a0 2024-01-23T08:45:16.077 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.078+0000 7f6ad1705700 1 -- 172.21.15.137:0/4045695746 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6acc0fb6b0 con 0x7f6acc0fd5a0 2024-01-23T08:45:16.077 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.078+0000 7f6ad1705700 1 -- 172.21.15.137:0/4045695746 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6acc0fbae0 con 0x7f6acc0fd5a0 2024-01-23T08:45:16.077 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.079+0000 7f6ab37fe700 1 -- 172.21.15.137:0/4045695746 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7f6abc01c820 con 0x7f6acc0fd5a0 2024-01-23T08:45:16.078 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.079+0000 7f6ab37fe700 1 --2- 172.21.15.137:0/4045695746 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f6ab4036810 0x7f6ab4038cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:16.078 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.079+0000 7f6ad1705700 1 -- 172.21.15.137:0/4045695746 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6acc1066f0 con 0x7f6acc0fd5a0 2024-01-23T08:45:16.078 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.079+0000 7f6ab37fe700 1 -- 172.21.15.137:0/4045695746 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f6abc049ab0 con 0x7f6acc0fd5a0 2024-01-23T08:45:16.078 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.079+0000 7f6aca7fc700 1 --2- 172.21.15.137:0/4045695746 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f6ab4036810 0x7f6ab4038cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:45:16.078 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.080+0000 7f6aca7fc700 1 --2- 172.21.15.137:0/4045695746 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f6ab4036810 0x7f6ab4038cc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0009990 tx=0x7f6ac0006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:45:16.082 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.084+0000 7f6ab37fe700 1 -- 172.21.15.137:0/4045695746 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6abc011dd0 con 0x7f6acc0fd5a0 2024-01-23T08:45:16.368 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.370+0000 7f6ad1705700 1 -- 172.21.15.137:0/4045695746 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6acc04bd60 con 0x7f6acc0fd5a0 2024-01-23T08:45:16.369 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.371+0000 7f6ab37fe700 1 -- 172.21.15.137:0/4045695746 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f6abc011370 con 0x7f6acc0fd5a0 2024-01-23T08:45:16.369 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:45:16.369 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"3fc07a38-b9cb-11ee-95b1-87774f69a715","modified":"2024-01-23T08:43:01.494526Z","created":"2024-01-23T08:43:01.494526Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-23T08:45:16.371 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.373+0000 7f6ad1705700 1 -- 172.21.15.137:0/4045695746 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f6ab4036810 msgr2=0x7f6ab4038cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:16.371 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.373+0000 7f6ad1705700 1 --2- 172.21.15.137:0/4045695746 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f6ab4036810 0x7f6ab4038cc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0009990 tx=0x7f6ac0006e40 comp rx=0 tx=0).stop 2024-01-23T08:45:16.371 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.373+0000 7f6ad1705700 1 -- 172.21.15.137:0/4045695746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6acc0fd5a0 msgr2=0x7f6acc0fd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:16.371 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.373+0000 7f6ad1705700 1 --2- 172.21.15.137:0/4045695746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6acc0fd5a0 0x7f6acc0fd030 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6abc009040 tx=0x7f6abc009e30 comp rx=0 tx=0).stop 2024-01-23T08:45:16.371 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.373+0000 7f6ad1705700 1 -- 172.21.15.137:0/4045695746 shutdown_connections 2024-01-23T08:45:16.372 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.373+0000 7f6ad1705700 1 --2- 172.21.15.137:0/4045695746 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f6ab4036810 0x7f6ab4038cc0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:16.372 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.373+0000 7f6ad1705700 1 --2- 172.21.15.137:0/4045695746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6acc0fd5a0 0x7f6acc0fd030 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:16.372 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.373+0000 7f6ad1705700 1 -- 172.21.15.137:0/4045695746 >> 172.21.15.137:0/4045695746 conn(0x7f6acc0f7020 msgr2=0x7f6acc0f7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:45:16.372 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.373+0000 7f6ad1705700 1 -- 172.21.15.137:0/4045695746 shutdown_connections 2024-01-23T08:45:16.372 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:16.373+0000 7f6ad1705700 1 -- 172.21.15.137:0/4045695746 wait complete. 2024-01-23T08:45:16.372 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-01-23T08:45:17.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:16 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:17.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:16 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:17.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:16 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:17.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:16 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:17.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:16 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd/host:smithi137", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:45:17.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:16 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:45:17.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:16 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:45:17.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:16 smithi059 ceph-mon[28424]: Updating smithi059:/etc/ceph/ceph.conf 2024-01-23T08:45:17.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:16 smithi059 ceph-mon[28424]: Updating smithi137:/etc/ceph/ceph.conf 2024-01-23T08:45:17.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:16 smithi059 ceph-mon[28424]: Updating smithi137:/var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-23T08:45:17.223 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:16 smithi059 ceph-mon[28424]: Updating smithi059:/var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-23T08:45:17.223 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:16 smithi059 ceph-mon[28424]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-01-23T08:45:17.223 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:16 smithi059 ceph-mon[28424]: from='client.? 172.21.15.137:0/4045695746' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:17.223 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:16 smithi059 ceph-mon[28424]: Updating smithi059:/etc/ceph/ceph.client.admin.keyring 2024-01-23T08:45:17.223 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:16 smithi059 ceph-mon[28424]: Updating smithi137:/var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/config/ceph.client.admin.keyring 2024-01-23T08:45:17.223 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:16 smithi059 ceph-mon[28424]: Updating smithi059:/var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/config/ceph.client.admin.keyring 2024-01-23T08:45:17.223 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:16 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:17.223 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:16 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:17.223 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:16 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:17.223 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:16 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:17.223 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:16 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:17.223 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:16 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi137", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-23T08:45:17.223 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:16 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi137", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-23T08:45:17.223 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:16 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:45:17.987 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-23T08:45:17.987 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph mon dump -f json 2024-01-23T08:45:18.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:17 smithi059 ceph-mon[28424]: Deploying daemon crash.smithi137 on smithi137 2024-01-23T08:45:18.228 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-23T08:45:19.913 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.914+0000 7faf98474700 1 -- 172.21.15.137:0/2075230511 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf90102ab0 msgr2=0x7faf90102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:19.913 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.914+0000 7faf98474700 1 --2- 172.21.15.137:0/2075230511 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf90102ab0 0x7faf90102e80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7faf8c0048c0 tx=0x7faf8c009e30 comp rx=0 tx=0).stop 2024-01-23T08:45:19.913 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.914+0000 7faf98474700 1 -- 172.21.15.137:0/2075230511 shutdown_connections 2024-01-23T08:45:19.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.914+0000 7faf98474700 1 --2- 172.21.15.137:0/2075230511 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf90102ab0 0x7faf90102e80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:19.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.914+0000 7faf98474700 1 -- 172.21.15.137:0/2075230511 >> 172.21.15.137:0/2075230511 conn(0x7faf900f85f0 msgr2=0x7faf900faa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:45:19.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.914+0000 7faf98474700 1 -- 172.21.15.137:0/2075230511 shutdown_connections 2024-01-23T08:45:19.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.914+0000 7faf98474700 1 -- 172.21.15.137:0/2075230511 wait complete. 2024-01-23T08:45:19.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.915+0000 7faf98474700 1 Processor -- start 2024-01-23T08:45:19.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.915+0000 7faf98474700 1 -- start start 2024-01-23T08:45:19.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.915+0000 7faf98474700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf90102ab0 0x7faf90112d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:19.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.915+0000 7faf98474700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf90113280 con 0x7faf90102ab0 2024-01-23T08:45:19.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.915+0000 7faf96210700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf90102ab0 0x7faf90112d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:45:19.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.915+0000 7faf96210700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf90102ab0 0x7faf90112d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.137:47434/0 (socket says 172.21.15.137:47434) 2024-01-23T08:45:19.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.915+0000 7faf96210700 1 -- 172.21.15.137:0/1296824986 learned_addr learned my addr 172.21.15.137:0/1296824986 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-23T08:45:19.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.916+0000 7faf96210700 1 -- 172.21.15.137:0/1296824986 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf8c005040 con 0x7faf90102ab0 2024-01-23T08:45:19.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.916+0000 7faf96210700 1 --2- 172.21.15.137:0/1296824986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf90102ab0 0x7faf90112d40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7faf8c009300 tx=0x7faf8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:45:19.915 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.916+0000 7faf86ffd700 1 -- 172.21.15.137:0/1296824986 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf8c004e60 con 0x7faf90102ab0 2024-01-23T08:45:19.915 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.916+0000 7faf98474700 1 -- 172.21.15.137:0/1296824986 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf90113480 con 0x7faf90102ab0 2024-01-23T08:45:19.915 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.916+0000 7faf86ffd700 1 -- 172.21.15.137:0/1296824986 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf8c013420 con 0x7faf90102ab0 2024-01-23T08:45:19.915 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.916+0000 7faf86ffd700 1 -- 172.21.15.137:0/1296824986 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf8c01c5b0 con 0x7faf90102ab0 2024-01-23T08:45:19.915 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.916+0000 7faf98474700 1 -- 172.21.15.137:0/1296824986 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf90113970 con 0x7faf90102ab0 2024-01-23T08:45:19.916 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.917+0000 7faf86ffd700 1 -- 172.21.15.137:0/1296824986 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7faf8c01c7f0 con 0x7faf90102ab0 2024-01-23T08:45:19.916 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.917+0000 7faf86ffd700 1 --2- 172.21.15.137:0/1296824986 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7faf7c0367c0 0x7faf7c038c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:19.916 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.917+0000 7faf86ffd700 1 -- 172.21.15.137:0/1296824986 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7faf8c049980 con 0x7faf90102ab0 2024-01-23T08:45:19.916 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.917+0000 7faf98474700 1 -- 172.21.15.137:0/1296824986 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf9005f450 con 0x7faf90102ab0 2024-01-23T08:45:19.916 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.917+0000 7faf95a0f700 1 --2- 172.21.15.137:0/1296824986 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7faf7c0367c0 0x7faf7c038c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:45:19.916 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.918+0000 7faf95a0f700 1 --2- 172.21.15.137:0/1296824986 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7faf7c0367c0 0x7faf7c038c70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7faf80009990 tx=0x7faf80006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:45:19.919 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:19.922+0000 7faf86ffd700 1 -- 172.21.15.137:0/1296824986 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7faf8c023300 con 0x7faf90102ab0 2024-01-23T08:45:20.184 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:20.187+0000 7faf98474700 1 -- 172.21.15.137:0/1296824986 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf90101740 con 0x7faf90102ab0 2024-01-23T08:45:20.185 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:20.188+0000 7faf86ffd700 1 -- 172.21.15.137:0/1296824986 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7faf8c011b60 con 0x7faf90102ab0 2024-01-23T08:45:20.185 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:45:20.185 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"3fc07a38-b9cb-11ee-95b1-87774f69a715","modified":"2024-01-23T08:43:01.494526Z","created":"2024-01-23T08:43:01.494526Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-23T08:45:20.187 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:20.190+0000 7faf98474700 1 -- 172.21.15.137:0/1296824986 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7faf7c0367c0 msgr2=0x7faf7c038c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:20.187 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:20.190+0000 7faf98474700 1 --2- 172.21.15.137:0/1296824986 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7faf7c0367c0 0x7faf7c038c70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7faf80009990 tx=0x7faf80006e40 comp rx=0 tx=0).stop 2024-01-23T08:45:20.187 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:20.190+0000 7faf98474700 1 -- 172.21.15.137:0/1296824986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf90102ab0 msgr2=0x7faf90112d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:20.187 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:20.190+0000 7faf98474700 1 --2- 172.21.15.137:0/1296824986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf90102ab0 0x7faf90112d40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7faf8c009300 tx=0x7faf8c009e30 comp rx=0 tx=0).stop 2024-01-23T08:45:20.187 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:20.190+0000 7faf98474700 1 -- 172.21.15.137:0/1296824986 shutdown_connections 2024-01-23T08:45:20.187 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:20.190+0000 7faf98474700 1 --2- 172.21.15.137:0/1296824986 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7faf7c0367c0 0x7faf7c038c70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:20.187 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:20.190+0000 7faf98474700 1 --2- 172.21.15.137:0/1296824986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf90102ab0 0x7faf90112d40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:20.187 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:20.190+0000 7faf98474700 1 -- 172.21.15.137:0/1296824986 >> 172.21.15.137:0/1296824986 conn(0x7faf900f85f0 msgr2=0x7faf900fa580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:45:20.187 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:20.190+0000 7faf98474700 1 -- 172.21.15.137:0/1296824986 shutdown_connections 2024-01-23T08:45:20.187 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:20.190+0000 7faf98474700 1 -- 172.21.15.137:0/1296824986 wait complete. 2024-01-23T08:45:20.188 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-01-23T08:45:21.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:20 smithi059 ceph-mon[28424]: from='client.? 172.21.15.137:0/1296824986' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:21.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:20 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:21.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:20 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:21.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:20 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:21.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:20 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:21.801 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-23T08:45:21.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph mon dump -f json 2024-01-23T08:45:22.047 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-23T08:45:22.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:21 smithi059 ceph-mon[28424]: Deploying daemon node-exporter.smithi137 on smithi137 2024-01-23T08:45:24.122 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.124+0000 7f3616c8c700 1 -- 172.21.15.137:0/4039306479 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36101019b0 msgr2=0x7f3610101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:24.122 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.124+0000 7f3616c8c700 1 --2- 172.21.15.137:0/4039306479 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36101019b0 0x7f3610101d80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f36040048c0 tx=0x7f3604009e30 comp rx=0 tx=0).stop 2024-01-23T08:45:24.122 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.124+0000 7f3616c8c700 1 -- 172.21.15.137:0/4039306479 shutdown_connections 2024-01-23T08:45:24.122 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.124+0000 7f3616c8c700 1 --2- 172.21.15.137:0/4039306479 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36101019b0 0x7f3610101d80 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:24.122 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.124+0000 7f3616c8c700 1 -- 172.21.15.137:0/4039306479 >> 172.21.15.137:0/4039306479 conn(0x7f36100f7530 msgr2=0x7f36100f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:45:24.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.124+0000 7f3616c8c700 1 -- 172.21.15.137:0/4039306479 shutdown_connections 2024-01-23T08:45:24.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.124+0000 7f3616c8c700 1 -- 172.21.15.137:0/4039306479 wait complete. 2024-01-23T08:45:24.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.124+0000 7f3616c8c700 1 Processor -- start 2024-01-23T08:45:24.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.125+0000 7f3616c8c700 1 -- start start 2024-01-23T08:45:24.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.125+0000 7f3616c8c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36101019b0 0x7f361010ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:24.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.125+0000 7f3616c8c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361010e330 con 0x7f36101019b0 2024-01-23T08:45:24.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.125+0000 7f3614a28700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36101019b0 0x7f361010ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:45:24.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.125+0000 7f3614a28700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36101019b0 0x7f361010ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.137:53022/0 (socket says 172.21.15.137:53022) 2024-01-23T08:45:24.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.125+0000 7f3614a28700 1 -- 172.21.15.137:0/520688025 learned_addr learned my addr 172.21.15.137:0/520688025 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-23T08:45:24.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.125+0000 7f3614a28700 1 -- 172.21.15.137:0/520688025 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3604005040 con 0x7f36101019b0 2024-01-23T08:45:24.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.126+0000 7f3614a28700 1 --2- 172.21.15.137:0/520688025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36101019b0 0x7f361010ddf0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3604009300 tx=0x7f3604009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:45:24.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.126+0000 7f360d7fa700 1 -- 172.21.15.137:0/520688025 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3604004e60 con 0x7f36101019b0 2024-01-23T08:45:24.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.126+0000 7f3616c8c700 1 -- 172.21.15.137:0/520688025 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f361010e530 con 0x7f36101019b0 2024-01-23T08:45:24.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.126+0000 7f360d7fa700 1 -- 172.21.15.137:0/520688025 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3604013420 con 0x7f36101019b0 2024-01-23T08:45:24.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.126+0000 7f360d7fa700 1 -- 172.21.15.137:0/520688025 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f360401c5b0 con 0x7f36101019b0 2024-01-23T08:45:24.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.126+0000 7f3616c8c700 1 -- 172.21.15.137:0/520688025 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f361010ea20 con 0x7f36101019b0 2024-01-23T08:45:24.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.127+0000 7f360d7fa700 1 -- 172.21.15.137:0/520688025 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7f3604009600 con 0x7f36101019b0 2024-01-23T08:45:24.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.127+0000 7f3616c8c700 1 -- 172.21.15.137:0/520688025 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3610100c20 con 0x7f36101019b0 2024-01-23T08:45:24.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.127+0000 7f360d7fa700 1 --2- 172.21.15.137:0/520688025 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f36000367c0 0x7f3600038c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:24.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.127+0000 7f360d7fa700 1 -- 172.21.15.137:0/520688025 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f3604049980 con 0x7f36101019b0 2024-01-23T08:45:24.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.127+0000 7f360ffff700 1 --2- 172.21.15.137:0/520688025 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f36000367c0 0x7f3600038c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:45:24.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.128+0000 7f360ffff700 1 --2- 172.21.15.137:0/520688025 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f36000367c0 0x7f3600038c70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f35fc009990 tx=0x7f35fc006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:45:24.129 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.132+0000 7f360d7fa700 1 -- 172.21.15.137:0/520688025 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3604025400 con 0x7f36101019b0 2024-01-23T08:45:24.398 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.402+0000 7f3616c8c700 1 -- 172.21.15.137:0/520688025 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f361004bd90 con 0x7f36101019b0 2024-01-23T08:45:24.399 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.402+0000 7f360d7fa700 1 -- 172.21.15.137:0/520688025 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f3604011370 con 0x7f36101019b0 2024-01-23T08:45:24.399 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:45:24.399 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"3fc07a38-b9cb-11ee-95b1-87774f69a715","modified":"2024-01-23T08:43:01.494526Z","created":"2024-01-23T08:43:01.494526Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-23T08:45:24.401 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.404+0000 7f3616c8c700 1 -- 172.21.15.137:0/520688025 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f36000367c0 msgr2=0x7f3600038c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:24.401 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.404+0000 7f3616c8c700 1 --2- 172.21.15.137:0/520688025 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f36000367c0 0x7f3600038c70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f35fc009990 tx=0x7f35fc006e40 comp rx=0 tx=0).stop 2024-01-23T08:45:24.401 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.404+0000 7f3616c8c700 1 -- 172.21.15.137:0/520688025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36101019b0 msgr2=0x7f361010ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:24.401 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.404+0000 7f3616c8c700 1 --2- 172.21.15.137:0/520688025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36101019b0 0x7f361010ddf0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3604009300 tx=0x7f3604009e30 comp rx=0 tx=0).stop 2024-01-23T08:45:24.401 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.404+0000 7f3616c8c700 1 -- 172.21.15.137:0/520688025 shutdown_connections 2024-01-23T08:45:24.401 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.404+0000 7f3616c8c700 1 --2- 172.21.15.137:0/520688025 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f36000367c0 0x7f3600038c70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:24.402 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.404+0000 7f3616c8c700 1 --2- 172.21.15.137:0/520688025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36101019b0 0x7f361010ddf0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:24.402 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.404+0000 7f3616c8c700 1 -- 172.21.15.137:0/520688025 >> 172.21.15.137:0/520688025 conn(0x7f36100f7530 msgr2=0x7f36100f8190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:45:24.402 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.404+0000 7f3616c8c700 1 -- 172.21.15.137:0/520688025 shutdown_connections 2024-01-23T08:45:24.402 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:24.405+0000 7f3616c8c700 1 -- 172.21.15.137:0/520688025 wait complete. 2024-01-23T08:45:24.402 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-01-23T08:45:24.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:24 smithi059 ceph-mon[28424]: from='client.? 172.21.15.137:0/520688025' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:26.099 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-23T08:45:26.099 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph mon dump -f json 2024-01-23T08:45:26.343 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-23T08:45:26.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:26 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:26.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:26 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:45:28.297 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.300+0000 7f7dfe1f7700 1 -- 172.21.15.137:0/2171679823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7df8102a90 msgr2=0x7f7df8102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:28.297 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.300+0000 7f7dfe1f7700 1 --2- 172.21.15.137:0/2171679823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7df8102a90 0x7f7df8102e60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7de80048c0 tx=0x7f7de8009e30 comp rx=0 tx=0).stop 2024-01-23T08:45:28.297 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.300+0000 7f7dfe1f7700 1 -- 172.21.15.137:0/2171679823 shutdown_connections 2024-01-23T08:45:28.297 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.300+0000 7f7dfe1f7700 1 --2- 172.21.15.137:0/2171679823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7df8102a90 0x7f7df8102e60 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:28.297 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.300+0000 7f7dfe1f7700 1 -- 172.21.15.137:0/2171679823 >> 172.21.15.137:0/2171679823 conn(0x7f7df80f85f0 msgr2=0x7f7df80faa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:45:28.298 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.300+0000 7f7dfe1f7700 1 -- 172.21.15.137:0/2171679823 shutdown_connections 2024-01-23T08:45:28.298 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.300+0000 7f7dfe1f7700 1 -- 172.21.15.137:0/2171679823 wait complete. 2024-01-23T08:45:28.299 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.301+0000 7f7dfe1f7700 1 Processor -- start 2024-01-23T08:45:28.299 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.301+0000 7f7dfe1f7700 1 -- start start 2024-01-23T08:45:28.299 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.301+0000 7f7dfe1f7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7df8102a90 0x7f7df8112d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:28.299 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.301+0000 7f7dfe1f7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df8113280 con 0x7f7df8102a90 2024-01-23T08:45:28.299 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.301+0000 7f7df77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7df8102a90 0x7f7df8112d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:45:28.299 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.301+0000 7f7df77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7df8102a90 0x7f7df8112d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.137:53036/0 (socket says 172.21.15.137:53036) 2024-01-23T08:45:28.299 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.301+0000 7f7df77fe700 1 -- 172.21.15.137:0/205195856 learned_addr learned my addr 172.21.15.137:0/205195856 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-23T08:45:28.299 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.302+0000 7f7df77fe700 1 -- 172.21.15.137:0/205195856 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de8005040 con 0x7f7df8102a90 2024-01-23T08:45:28.300 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.302+0000 7f7df77fe700 1 --2- 172.21.15.137:0/205195856 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7df8102a90 0x7f7df8112d40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7de8009300 tx=0x7f7de8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:45:28.300 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.302+0000 7f7df4ff9700 1 -- 172.21.15.137:0/205195856 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7de8004e60 con 0x7f7df8102a90 2024-01-23T08:45:28.300 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.302+0000 7f7dfe1f7700 1 -- 172.21.15.137:0/205195856 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7df8113480 con 0x7f7df8102a90 2024-01-23T08:45:28.300 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.302+0000 7f7dfe1f7700 1 -- 172.21.15.137:0/205195856 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7df8113970 con 0x7f7df8102a90 2024-01-23T08:45:28.300 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.302+0000 7f7df4ff9700 1 -- 172.21.15.137:0/205195856 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7de8013420 con 0x7f7df8102a90 2024-01-23T08:45:28.300 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.302+0000 7f7df4ff9700 1 -- 172.21.15.137:0/205195856 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7de80255c0 con 0x7f7df8102a90 2024-01-23T08:45:28.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.303+0000 7f7df4ff9700 1 -- 172.21.15.137:0/205195856 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7f7de801c3f0 con 0x7f7df8102a90 2024-01-23T08:45:28.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.303+0000 7f7dfe1f7700 1 -- 172.21.15.137:0/205195856 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7df805f450 con 0x7f7df8102a90 2024-01-23T08:45:28.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.303+0000 7f7df4ff9700 1 --2- 172.21.15.137:0/205195856 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f7de4036770 0x7f7de4038c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:28.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.303+0000 7f7df4ff9700 1 -- 172.21.15.137:0/205195856 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f7de8049ca0 con 0x7f7df8102a90 2024-01-23T08:45:28.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.303+0000 7f7deeffd700 1 --2- 172.21.15.137:0/205195856 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f7de4036770 0x7f7de4038c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:45:28.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.304+0000 7f7deeffd700 1 --2- 172.21.15.137:0/205195856 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f7de4036770 0x7f7de4038c20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7de0009990 tx=0x7f7de0006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:45:28.305 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.308+0000 7f7df4ff9700 1 -- 172.21.15.137:0/205195856 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7de801cc20 con 0x7f7df8102a90 2024-01-23T08:45:28.580 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.583+0000 7f7dfe1f7700 1 -- 172.21.15.137:0/205195856 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7df810e4a0 con 0x7f7df8102a90 2024-01-23T08:45:28.581 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.584+0000 7f7df4ff9700 1 -- 172.21.15.137:0/205195856 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f7de8011370 con 0x7f7df8102a90 2024-01-23T08:45:28.581 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:45:28.582 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"3fc07a38-b9cb-11ee-95b1-87774f69a715","modified":"2024-01-23T08:43:01.494526Z","created":"2024-01-23T08:43:01.494526Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-23T08:45:28.583 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.586+0000 7f7dfe1f7700 1 -- 172.21.15.137:0/205195856 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f7de4036770 msgr2=0x7f7de4038c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:28.583 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.586+0000 7f7dfe1f7700 1 --2- 172.21.15.137:0/205195856 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f7de4036770 0x7f7de4038c20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7de0009990 tx=0x7f7de0006e40 comp rx=0 tx=0).stop 2024-01-23T08:45:28.583 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.586+0000 7f7dfe1f7700 1 -- 172.21.15.137:0/205195856 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7df8102a90 msgr2=0x7f7df8112d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:28.584 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.586+0000 7f7dfe1f7700 1 --2- 172.21.15.137:0/205195856 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7df8102a90 0x7f7df8112d40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7de8009300 tx=0x7f7de8009e30 comp rx=0 tx=0).stop 2024-01-23T08:45:28.584 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.586+0000 7f7dfe1f7700 1 -- 172.21.15.137:0/205195856 shutdown_connections 2024-01-23T08:45:28.584 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.586+0000 7f7dfe1f7700 1 --2- 172.21.15.137:0/205195856 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f7de4036770 0x7f7de4038c20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:28.584 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.586+0000 7f7dfe1f7700 1 --2- 172.21.15.137:0/205195856 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7df8102a90 0x7f7df8112d40 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:28.584 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.586+0000 7f7dfe1f7700 1 -- 172.21.15.137:0/205195856 >> 172.21.15.137:0/205195856 conn(0x7f7df80f85f0 msgr2=0x7f7df80f9250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:45:28.584 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.587+0000 7f7dfe1f7700 1 -- 172.21.15.137:0/205195856 shutdown_connections 2024-01-23T08:45:28.584 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:28.587+0000 7f7dfe1f7700 1 -- 172.21.15.137:0/205195856 wait complete. 2024-01-23T08:45:28.584 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-01-23T08:45:29.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:28 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:29.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:28 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:29.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:28 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:29.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:28 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:29.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:28 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.ooirrh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-23T08:45:29.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:28 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.ooirrh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-01-23T08:45:29.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:28 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-23T08:45:29.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:28 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:45:29.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:28 smithi059 ceph-mon[28424]: Deploying daemon mgr.smithi137.ooirrh on smithi137 2024-01-23T08:45:29.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:28 smithi059 ceph-mon[28424]: from='client.? 172.21.15.137:0/205195856' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:30.621 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-23T08:45:30.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph mon dump -f json 2024-01-23T08:45:30.886 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-23T08:45:31.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:31 smithi059 ceph-mon[28424]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:31.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:31 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:31.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:31 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:31.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:31 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:31.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:31 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:31.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:31 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:31.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:31 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-23T08:45:31.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:31 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:45:32.675 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.527+0000 7fd2cce45700 1 -- 172.21.15.137:0/1384663885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c80fda70 msgr2=0x7fd2c80fde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:32.675 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.527+0000 7fd2cce45700 1 --2- 172.21.15.137:0/1384663885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c80fda70 0x7fd2c80fde40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd2b80048c0 tx=0x7fd2b8009e30 comp rx=0 tx=0).stop 2024-01-23T08:45:32.675 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.527+0000 7fd2cce45700 1 -- 172.21.15.137:0/1384663885 shutdown_connections 2024-01-23T08:45:32.675 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.527+0000 7fd2cce45700 1 --2- 172.21.15.137:0/1384663885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c80fda70 0x7fd2c80fde40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:32.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.527+0000 7fd2cce45700 1 -- 172.21.15.137:0/1384663885 >> 172.21.15.137:0/1384663885 conn(0x7fd2c80726e0 msgr2=0x7fd2c8072ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:45:32.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.527+0000 7fd2cce45700 1 -- 172.21.15.137:0/1384663885 shutdown_connections 2024-01-23T08:45:32.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.527+0000 7fd2cce45700 1 -- 172.21.15.137:0/1384663885 wait complete. 2024-01-23T08:45:32.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.527+0000 7fd2cce45700 1 Processor -- start 2024-01-23T08:45:32.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.527+0000 7fd2cce45700 1 -- start start 2024-01-23T08:45:32.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.527+0000 7fd2cce45700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c80fda70 0x7fd2c8112d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:32.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.527+0000 7fd2cce45700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c81132d0 con 0x7fd2c80fda70 2024-01-23T08:45:32.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.528+0000 7fd2c659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c80fda70 0x7fd2c8112d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:45:32.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.528+0000 7fd2c659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c80fda70 0x7fd2c8112d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.137:42164/0 (socket says 172.21.15.137:42164) 2024-01-23T08:45:32.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.528+0000 7fd2c659c700 1 -- 172.21.15.137:0/1386400185 learned_addr learned my addr 172.21.15.137:0/1386400185 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-23T08:45:32.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.528+0000 7fd2c659c700 1 -- 172.21.15.137:0/1386400185 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b8005040 con 0x7fd2c80fda70 2024-01-23T08:45:32.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.528+0000 7fd2c659c700 1 --2- 172.21.15.137:0/1386400185 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c80fda70 0x7fd2c8112d90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8009300 tx=0x7fd2b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:45:32.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.528+0000 7fd2b6ffd700 1 -- 172.21.15.137:0/1386400185 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2b8004e60 con 0x7fd2c80fda70 2024-01-23T08:45:32.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.528+0000 7fd2b6ffd700 1 -- 172.21.15.137:0/1386400185 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd2b8013420 con 0x7fd2c80fda70 2024-01-23T08:45:32.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.528+0000 7fd2b6ffd700 1 -- 172.21.15.137:0/1386400185 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2b801c5b0 con 0x7fd2c80fda70 2024-01-23T08:45:32.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.528+0000 7fd2cce45700 1 -- 172.21.15.137:0/1386400185 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c81134d0 con 0x7fd2c80fda70 2024-01-23T08:45:32.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.529+0000 7fd2cce45700 1 -- 172.21.15.137:0/1386400185 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c8113940 con 0x7fd2c80fda70 2024-01-23T08:45:32.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.529+0000 7fd2b4ff9700 1 -- 172.21.15.137:0/1386400185 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2ac0052b0 con 0x7fd2c80fda70 2024-01-23T08:45:32.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.529+0000 7fd2b6ffd700 1 -- 172.21.15.137:0/1386400185 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7fd2b801c7f0 con 0x7fd2c80fda70 2024-01-23T08:45:32.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.530+0000 7fd2b6ffd700 1 --2- 172.21.15.137:0/1386400185 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fd2b00367c0 0x7fd2b0038c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:32.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.530+0000 7fd2b6ffd700 1 -- 172.21.15.137:0/1386400185 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fd2b8049a50 con 0x7fd2c80fda70 2024-01-23T08:45:32.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.530+0000 7fd2c5d9b700 1 --2- 172.21.15.137:0/1386400185 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fd2b00367c0 0x7fd2b0038c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:45:32.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.530+0000 7fd2c5d9b700 1 --2- 172.21.15.137:0/1386400185 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fd2b00367c0 0x7fd2b0038c70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc009990 tx=0x7fd2bc006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:45:32.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.535+0000 7fd2b6ffd700 1 -- 172.21.15.137:0/1386400185 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fd2b8023300 con 0x7fd2c80fda70 2024-01-23T08:45:32.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:32 smithi059 ceph-mon[28424]: Deploying daemon mon.smithi137 on smithi137 2024-01-23T08:45:32.806 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.809+0000 7fd2b4ff9700 1 -- 172.21.15.137:0/1386400185 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd2ac005560 con 0x7fd2c80fda70 2024-01-23T08:45:32.807 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.810+0000 7fd2b6ffd700 1 -- 172.21.15.137:0/1386400185 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fd2b8011b60 con 0x7fd2c80fda70 2024-01-23T08:45:32.807 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:45:32.807 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"3fc07a38-b9cb-11ee-95b1-87774f69a715","modified":"2024-01-23T08:43:01.494526Z","created":"2024-01-23T08:43:01.494526Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-23T08:45:32.809 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.812+0000 7fd2b4ff9700 1 -- 172.21.15.137:0/1386400185 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fd2b00367c0 msgr2=0x7fd2b0038c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:32.809 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.812+0000 7fd2b4ff9700 1 --2- 172.21.15.137:0/1386400185 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fd2b00367c0 0x7fd2b0038c70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc009990 tx=0x7fd2bc006e40 comp rx=0 tx=0).stop 2024-01-23T08:45:32.809 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.812+0000 7fd2b4ff9700 1 -- 172.21.15.137:0/1386400185 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c80fda70 msgr2=0x7fd2c8112d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:32.809 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.812+0000 7fd2b4ff9700 1 --2- 172.21.15.137:0/1386400185 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c80fda70 0x7fd2c8112d90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8009300 tx=0x7fd2b8009e30 comp rx=0 tx=0).stop 2024-01-23T08:45:32.809 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.812+0000 7fd2b4ff9700 1 -- 172.21.15.137:0/1386400185 shutdown_connections 2024-01-23T08:45:32.809 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.812+0000 7fd2b4ff9700 1 --2- 172.21.15.137:0/1386400185 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fd2b00367c0 0x7fd2b0038c70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:32.809 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.812+0000 7fd2b4ff9700 1 --2- 172.21.15.137:0/1386400185 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2c80fda70 0x7fd2c8112d90 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:32.809 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.812+0000 7fd2b4ff9700 1 -- 172.21.15.137:0/1386400185 >> 172.21.15.137:0/1386400185 conn(0x7fd2c80726e0 msgr2=0x7fd2c8101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:45:32.809 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.812+0000 7fd2b4ff9700 1 -- 172.21.15.137:0/1386400185 shutdown_connections 2024-01-23T08:45:32.809 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:32.812+0000 7fd2b4ff9700 1 -- 172.21.15.137:0/1386400185 wait complete. 2024-01-23T08:45:32.810 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-01-23T08:45:33.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:33 smithi059 ceph-mon[28424]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:33.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:33 smithi059 ceph-mon[28424]: from='client.? 172.21.15.137:0/1386400185' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:34.731 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-23T08:45:34.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph mon dump -f json 2024-01-23T08:45:35.089 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-23T08:45:35.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:35 smithi059 ceph-mon[28424]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:35.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:35 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:35.900 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:35 smithi137 systemd[1]: Starting Ceph mon.smithi137 for 3fc07a38-b9cb-11ee-95b1-87774f69a715... 2024-01-23T08:45:37.106 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 podman[35735]: 2024-01-23 08:45:37.011772153 +0000 UTC m=+0.290064423 container create e1d4c945bf497c86dd42103b5aaa357d1975b38171b1e8a4dbfed858b29824e9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi137, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.31.3, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-b824e5b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240118, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e) 2024-01-23T08:45:37.106 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 podman[35735]: 2024-01-23 08:45:36.918729072 +0000 UTC m=+0.197021342 image pull 61b360e6ff650187c031550cf8bb1c0d3e28fc466cffb29cb06fa0da2544d680 quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001 2024-01-23T08:45:37.287 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.288+0000 7ff276a6b700 1 -- 172.21.15.137:0/3598836945 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff270102ab0 msgr2=0x7ff270102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:37.287 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.288+0000 7ff276a6b700 1 --2- 172.21.15.137:0/3598836945 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff270102ab0 0x7ff270102e80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff2640048c0 tx=0x7ff264009e30 comp rx=0 tx=0).stop 2024-01-23T08:45:37.287 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.288+0000 7ff276a6b700 1 -- 172.21.15.137:0/3598836945 shutdown_connections 2024-01-23T08:45:37.287 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.288+0000 7ff276a6b700 1 --2- 172.21.15.137:0/3598836945 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff270102ab0 0x7ff270102e80 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:37.287 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.288+0000 7ff276a6b700 1 -- 172.21.15.137:0/3598836945 >> 172.21.15.137:0/3598836945 conn(0x7ff2700f85f0 msgr2=0x7ff2700faa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:45:37.287 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.289+0000 7ff276a6b700 1 -- 172.21.15.137:0/3598836945 shutdown_connections 2024-01-23T08:45:37.287 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.289+0000 7ff276a6b700 1 -- 172.21.15.137:0/3598836945 wait complete. 2024-01-23T08:45:37.288 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.289+0000 7ff276a6b700 1 Processor -- start 2024-01-23T08:45:37.288 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.289+0000 7ff276a6b700 1 -- start start 2024-01-23T08:45:37.289 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.290+0000 7ff276a6b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff270102ab0 0x7ff270112d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:37.289 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.290+0000 7ff276a6b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff270113280 con 0x7ff270102ab0 2024-01-23T08:45:37.289 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.290+0000 7ff274807700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff270102ab0 0x7ff270112d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:45:37.289 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.290+0000 7ff274807700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff270102ab0 0x7ff270112d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.137:42190/0 (socket says 172.21.15.137:42190) 2024-01-23T08:45:37.289 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.290+0000 7ff274807700 1 -- 172.21.15.137:0/43023214 learned_addr learned my addr 172.21.15.137:0/43023214 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-23T08:45:37.289 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.290+0000 7ff274807700 1 -- 172.21.15.137:0/43023214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff264005040 con 0x7ff270102ab0 2024-01-23T08:45:37.289 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.290+0000 7ff274807700 1 --2- 172.21.15.137:0/43023214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff270102ab0 0x7ff270112d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff264009300 tx=0x7ff264009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:45:37.289 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.291+0000 7ff26d7fa700 1 -- 172.21.15.137:0/43023214 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff264004e60 con 0x7ff270102ab0 2024-01-23T08:45:37.289 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.291+0000 7ff26d7fa700 1 -- 172.21.15.137:0/43023214 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff264013420 con 0x7ff270102ab0 2024-01-23T08:45:37.289 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.291+0000 7ff26d7fa700 1 -- 172.21.15.137:0/43023214 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff26401c5b0 con 0x7ff270102ab0 2024-01-23T08:45:37.289 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.291+0000 7ff276a6b700 1 -- 172.21.15.137:0/43023214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff270113480 con 0x7ff270102ab0 2024-01-23T08:45:37.289 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.291+0000 7ff276a6b700 1 -- 172.21.15.137:0/43023214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff270113970 con 0x7ff270102ab0 2024-01-23T08:45:37.290 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.292+0000 7ff26d7fa700 1 -- 172.21.15.137:0/43023214 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7ff264009600 con 0x7ff270102ab0 2024-01-23T08:45:37.290 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.291+0000 7ff276a6b700 1 -- 172.21.15.137:0/43023214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff27005f450 con 0x7ff270102ab0 2024-01-23T08:45:37.290 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.292+0000 7ff26d7fa700 1 --2- 172.21.15.137:0/43023214 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7ff2600367c0 0x7ff260038c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:37.291 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.292+0000 7ff26d7fa700 1 -- 172.21.15.137:0/43023214 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7ff264049a10 con 0x7ff270102ab0 2024-01-23T08:45:37.291 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.292+0000 7ff26ffff700 1 --2- 172.21.15.137:0/43023214 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7ff2600367c0 0x7ff260038c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:45:37.291 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.293+0000 7ff26ffff700 1 --2- 172.21.15.137:0/43023214 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7ff2600367c0 0x7ff260038c70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff25c009990 tx=0x7ff25c006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:45:37.295 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.297+0000 7ff26d7fa700 1 -- 172.21.15.137:0/43023214 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ff264025400 con 0x7ff270102ab0 2024-01-23T08:45:37.423 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.424+0000 7ff26d7fa700 1 -- 172.21.15.137:0/43023214 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff26404e020 con 0x7ff270102ab0 2024-01-23T08:45:37.567 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:37.570+0000 7ff276a6b700 1 -- 172.21.15.137:0/43023214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff27010e4b0 con 0x7ff270102ab0 2024-01-23T08:45:37.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:37 smithi059 ceph-mon[28424]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 podman[35735]: 2024-01-23 08:45:37.364841465 +0000 UTC m=+0.643133736 container init e1d4c945bf497c86dd42103b5aaa357d1975b38171b1e8a4dbfed858b29824e9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi137, org.label-schema.build-date=20240118, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, io.buildah.version=1.31.3, ceph=True, org.label-schema.vendor=CentOS, RELEASE=quincy-b824e5b, GIT_CLEAN=True) 2024-01-23T08:45:37.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 podman[35735]: 2024-01-23 08:45:37.36809979 +0000 UTC m=+0.646392061 container start e1d4c945bf497c86dd42103b5aaa357d1975b38171b1e8a4dbfed858b29824e9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi137, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240118, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.31.3, RELEASE=quincy-b824e5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-01-23T08:45:37.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: set uid:gid to 167:167 (ceph:ceph) 2024-01-23T08:45:37.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: ceph version 17.2.7-397-gb824e5be (b824e5befce1d427fda2bc90ae52665f0c7e2770) quincy (stable), process ceph-mon, pid 2 2024-01-23T08:45:37.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pidfile_write: ignore empty --pid-file 2024-01-23T08:45:37.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: load: jerasure load: lrc 2024-01-23T08:45:37.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: RocksDB version: 6.15.5 2024-01-23T08:45:37.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-23T08:45:37.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Compile date Jan 22 2024 2024-01-23T08:45:37.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: DB SUMMARY 2024-01-23T08:45:37.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: DB Session ID: EBYT253LWZLDYCHBALES 2024-01-23T08:45:37.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: CURRENT file: CURRENT 2024-01-23T08:45:37.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: IDENTITY file: IDENTITY 2024-01-23T08:45:37.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-01-23T08:45:37.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi137/store.db dir, Total Num: 0, files: 2024-01-23T08:45:37.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi137/store.db: 000004.log size: 511 ; 2024-01-23T08:45:37.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.error_if_exists: 0 2024-01-23T08:45:37.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.create_if_missing: 0 2024-01-23T08:45:37.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.paranoid_checks: 1 2024-01-23T08:45:37.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-01-23T08:45:37.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.env: 0x562d305a0a80 2024-01-23T08:45:37.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.fs: Posix File System 2024-01-23T08:45:37.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.info_log: 0x562d32445de0 2024-01-23T08:45:37.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_file_opening_threads: 16 2024-01-23T08:45:37.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.statistics: (nil) 2024-01-23T08:45:37.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.use_fsync: 0 2024-01-23T08:45:37.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_log_file_size: 0 2024-01-23T08:45:37.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-23T08:45:37.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.log_file_time_to_roll: 0 2024-01-23T08:45:37.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.keep_log_file_num: 1000 2024-01-23T08:45:37.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.recycle_log_file_num: 0 2024-01-23T08:45:37.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.allow_fallocate: 1 2024-01-23T08:45:37.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.allow_mmap_reads: 0 2024-01-23T08:45:37.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.allow_mmap_writes: 0 2024-01-23T08:45:37.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.use_direct_reads: 0 2024-01-23T08:45:37.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-23T08:45:37.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.create_missing_column_families: 0 2024-01-23T08:45:37.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.db_log_dir: 2024-01-23T08:45:37.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi137/store.db 2024-01-23T08:45:37.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.table_cache_numshardbits: 6 2024-01-23T08:45:37.797 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.WAL_ttl_seconds: 0 2024-01-23T08:45:37.797 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.WAL_size_limit_MB: 0 2024-01-23T08:45:37.797 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-23T08:45:37.797 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-23T08:45:37.797 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.is_fd_close_on_exec: 1 2024-01-23T08:45:37.797 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.advise_random_on_open: 1 2024-01-23T08:45:37.797 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.db_write_buffer_size: 0 2024-01-23T08:45:37.797 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.write_buffer_manager: 0x562d331baa80 2024-01-23T08:45:37.797 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-23T08:45:37.797 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-23T08:45:37.797 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-23T08:45:37.797 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.use_adaptive_mutex: 0 2024-01-23T08:45:37.797 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.rate_limiter: (nil) 2024-01-23T08:45:37.797 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-23T08:45:37.797 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.wal_recovery_mode: 2 2024-01-23T08:45:37.798 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.enable_thread_tracking: 0 2024-01-23T08:45:37.798 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.enable_pipelined_write: 0 2024-01-23T08:45:37.798 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.unordered_write: 0 2024-01-23T08:45:37.798 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-23T08:45:37.798 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-23T08:45:37.798 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-23T08:45:37.798 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-23T08:45:37.798 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.row_cache: None 2024-01-23T08:45:37.798 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.wal_filter: None 2024-01-23T08:45:37.798 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-23T08:45:37.801 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.allow_ingest_behind: 0 2024-01-23T08:45:37.801 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.preserve_deletes: 0 2024-01-23T08:45:37.801 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.two_write_queues: 0 2024-01-23T08:45:37.801 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.manual_wal_flush: 0 2024-01-23T08:45:37.801 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.atomic_flush: 0 2024-01-23T08:45:37.801 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-23T08:45:37.801 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.persist_stats_to_disk: 0 2024-01-23T08:45:37.801 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.write_dbid_to_manifest: 0 2024-01-23T08:45:37.801 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.log_readahead_size: 0 2024-01-23T08:45:37.801 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-01-23T08:45:37.801 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.best_efforts_recovery: 0 2024-01-23T08:45:37.801 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-01-23T08:45:37.801 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-01-23T08:45:37.801 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.allow_data_in_errors: 0 2024-01-23T08:45:37.802 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.db_host_id: __hostname__ 2024-01-23T08:45:37.802 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_background_jobs: 2 2024-01-23T08:45:37.802 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_background_compactions: -1 2024-01-23T08:45:37.802 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_subcompactions: 1 2024-01-23T08:45:37.802 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-23T08:45:37.802 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-23T08:45:37.802 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.delayed_write_rate : 16777216 2024-01-23T08:45:37.802 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_total_wal_size: 0 2024-01-23T08:45:37.802 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-23T08:45:37.802 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.stats_dump_period_sec: 600 2024-01-23T08:45:37.802 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.stats_persist_period_sec: 600 2024-01-23T08:45:37.802 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-23T08:45:37.802 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_open_files: -1 2024-01-23T08:45:37.802 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.bytes_per_sync: 0 2024-01-23T08:45:37.803 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.wal_bytes_per_sync: 0 2024-01-23T08:45:37.803 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.strict_bytes_per_sync: 0 2024-01-23T08:45:37.803 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.compaction_readahead_size: 0 2024-01-23T08:45:37.803 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_background_flushes: -1 2024-01-23T08:45:37.803 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Compression algorithms supported: 2024-01-23T08:45:37.803 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-23T08:45:37.803 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: kZSTD supported: 0 2024-01-23T08:45:37.803 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: kXpressCompression supported: 0 2024-01-23T08:45:37.803 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: kLZ4HCCompression supported: 1 2024-01-23T08:45:37.803 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: kZlibCompression supported: 1 2024-01-23T08:45:37.803 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: kSnappyCompression supported: 1 2024-01-23T08:45:37.803 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: kLZ4Compression supported: 1 2024-01-23T08:45:37.803 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: kBZip2Compression supported: 0 2024-01-23T08:45:37.803 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Fast CRC32 supported: Supported on x86 2024-01-23T08:45:37.803 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi137/store.db/MANIFEST-000003 2024-01-23T08:45:37.804 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-01-23T08:45:37.804 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-23T08:45:37.804 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.merge_operator: 2024-01-23T08:45:37.804 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.compaction_filter: None 2024-01-23T08:45:37.804 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.compaction_filter_factory: None 2024-01-23T08:45:37.804 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.sst_partitioner_factory: None 2024-01-23T08:45:37.804 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.memtable_factory: SkipListFactory 2024-01-23T08:45:37.804 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.table_factory: BlockBasedTable 2024-01-23T08:45:37.804 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562d3249afd8) 2024-01-23T08:45:37.804 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: cache_index_and_filter_blocks: 1 2024-01-23T08:45:37.804 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-23T08:45:37.804 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-23T08:45:37.804 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: pin_top_level_index_and_filter: 1 2024-01-23T08:45:37.804 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: index_type: 0 2024-01-23T08:45:37.804 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: data_block_index_type: 0 2024-01-23T08:45:37.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: index_shortening: 1 2024-01-23T08:45:37.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: data_block_hash_table_util_ratio: 0.750000 2024-01-23T08:45:37.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: hash_index_allow_collision: 1 2024-01-23T08:45:37.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: checksum: 1 2024-01-23T08:45:37.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: no_block_cache: 0 2024-01-23T08:45:37.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_cache: 0x562d324cd090 2024-01-23T08:45:37.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_cache_name: BinnedLRUCache 2024-01-23T08:45:37.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_cache_options: 2024-01-23T08:45:37.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: capacity : 536870912 2024-01-23T08:45:37.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: num_shard_bits : 4 2024-01-23T08:45:37.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: strict_capacity_limit : 0 2024-01-23T08:45:37.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: high_pri_pool_ratio: 0.000 2024-01-23T08:45:37.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_cache_compressed: (nil) 2024-01-23T08:45:37.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: persistent_cache: (nil) 2024-01-23T08:45:37.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_size: 4096 2024-01-23T08:45:37.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_size_deviation: 10 2024-01-23T08:45:37.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_restart_interval: 16 2024-01-23T08:45:37.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: index_block_restart_interval: 1 2024-01-23T08:45:37.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: metadata_block_size: 4096 2024-01-23T08:45:37.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: partition_filters: 0 2024-01-23T08:45:37.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: use_delta_encoding: 1 2024-01-23T08:45:37.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-23T08:45:37.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: whole_key_filtering: 1 2024-01-23T08:45:37.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: verify_compression: 0 2024-01-23T08:45:37.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: read_amp_bytes_per_bit: 0 2024-01-23T08:45:37.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: format_version: 4 2024-01-23T08:45:37.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: enable_index_compression: 1 2024-01-23T08:45:37.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_align: 0 2024-01-23T08:45:37.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.write_buffer_size: 33554432 2024-01-23T08:45:37.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_write_buffer_number: 2 2024-01-23T08:45:37.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.compression: NoCompression 2024-01-23T08:45:37.807 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.bottommost_compression: Disabled 2024-01-23T08:45:37.807 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.prefix_extractor: nullptr 2024-01-23T08:45:37.807 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-23T08:45:37.807 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.num_levels: 7 2024-01-23T08:45:37.807 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-23T08:45:37.807 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-23T08:45:37.807 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-23T08:45:37.807 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-23T08:45:37.807 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-23T08:45:37.807 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-23T08:45:37.807 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-23T08:45:37.807 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-23T08:45:37.807 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-01-23T08:45:37.807 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-23T08:45:37.808 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.compression_opts.window_bits: -14 2024-01-23T08:45:37.808 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.compression_opts.level: 32767 2024-01-23T08:45:37.808 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.compression_opts.strategy: 0 2024-01-23T08:45:37.808 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-23T08:45:37.808 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-23T08:45:37.808 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-01-23T08:45:37.808 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.compression_opts.enabled: false 2024-01-23T08:45:37.808 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-23T08:45:37.808 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-23T08:45:37.808 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-23T08:45:37.808 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.target_file_size_base: 67108864 2024-01-23T08:45:37.808 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.target_file_size_multiplier: 1 2024-01-23T08:45:37.808 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-23T08:45:37.808 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-23T08:45:37.808 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-23T08:45:37.808 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-23T08:45:37.809 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-23T08:45:37.809 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-23T08:45:37.809 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-23T08:45:37.809 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-23T08:45:37.809 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-23T08:45:37.809 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-23T08:45:37.809 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-23T08:45:37.809 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-23T08:45:37.809 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.arena_block_size: 4194304 2024-01-23T08:45:37.809 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-23T08:45:37.809 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-23T08:45:37.809 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-23T08:45:37.809 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.disable_auto_compactions: 0 2024-01-23T08:45:37.809 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-23T08:45:37.809 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-23T08:45:37.809 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-23T08:45:37.809 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-23T08:45:37.810 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-23T08:45:37.810 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-23T08:45:37.810 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-23T08:45:37.810 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-23T08:45:37.810 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-23T08:45:37.810 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-23T08:45:37.810 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.table_properties_collectors: 2024-01-23T08:45:37.810 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.inplace_update_support: 0 2024-01-23T08:45:37.810 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.inplace_update_num_locks: 10000 2024-01-23T08:45:37.810 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-23T08:45:37.810 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-23T08:45:37.811 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.memtable_huge_page_size: 0 2024-01-23T08:45:37.811 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.bloom_locality: 0 2024-01-23T08:45:37.811 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.max_successive_merges: 0 2024-01-23T08:45:37.811 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.optimize_filters_for_hits: 0 2024-01-23T08:45:37.811 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.paranoid_file_checks: 0 2024-01-23T08:45:37.811 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.force_consistency_checks: 1 2024-01-23T08:45:37.811 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.report_bg_io_stats: 0 2024-01-23T08:45:37.811 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.ttl: 2592000 2024-01-23T08:45:37.811 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.periodic_compaction_seconds: 0 2024-01-23T08:45:37.811 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.enable_blob_files: false 2024-01-23T08:45:37.811 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.min_blob_size: 0 2024-01-23T08:45:37.811 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.blob_file_size: 268435456 2024-01-23T08:45:37.811 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.blob_compression_type: NoCompression 2024-01-23T08:45:37.811 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.enable_blob_garbage_collection: false 2024-01-23T08:45:37.811 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-01-23T08:45:37.811 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi137/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-23T08:45:37.812 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-01-23T08:45:37.812 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-01-23T08:45:37.812 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: EVENT_LOG_v1 {"time_micros": 1705999537394660, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-01-23T08:45:37.812 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-01-23T08:45:37.812 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-23T08:45:37.812 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: EVENT_LOG_v1 {"time_micros": 1705999537395079, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705999537, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "8fd0f259-3ef1-4b21-9c06-0f0d9724fc00", "db_session_id": "EBYT253LWZLDYCHBALES"}} 2024-01-23T08:45:37.812 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-01-23T08:45:37.812 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: EVENT_LOG_v1 {"time_micros": 1705999537395490, "job": 1, "event": "recovery_finished"} 2024-01-23T08:45:37.812 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi137/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-01-23T08:45:37.812 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x562d32461180 2024-01-23T08:45:37.812 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: DB pointer 0x562d32536000 2024-01-23T08:45:37.812 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-01-23T08:45:37.812 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-01-23T08:45:37.812 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ** DB Stats ** 2024-01-23T08:45:37.813 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-23T08:45:37.813 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-23T08:45:37.813 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-23T08:45:37.813 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-23T08:45:37.813 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-23T08:45:37.813 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-23T08:45:37.813 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-23T08:45:37.813 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-01-23T08:45:37.813 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ** Compaction Stats [default] ** 2024-01-23T08:45:37.813 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-23T08:45:37.813 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-23T08:45:37.813 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 2024-01-23T08:45:37.813 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 2024-01-23T08:45:37.813 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 2024-01-23T08:45:37.813 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-01-23T08:45:37.814 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ** Compaction Stats [default] ** 2024-01-23T08:45:37.814 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-23T08:45:37.814 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-23T08:45:37.814 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 2024-01-23T08:45:37.814 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-23T08:45:37.814 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-01-23T08:45:37.814 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-23T08:45:37.814 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-01-23T08:45:37.814 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-01-23T08:45:37.814 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: AddFile(Keys): cumulative 0, interval 0 2024-01-23T08:45:37.814 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Cumulative compaction: 0.00 GB write, 0.41 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-23T08:45:37.814 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Interval compaction: 0.00 GB write, 0.41 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-23T08:45:37.814 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-23T08:45:37.814 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-01-23T08:45:37.815 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ** File Read Latency Histogram By Level [default] ** 2024-01-23T08:45:37.815 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-01-23T08:45:37.815 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ** Compaction Stats [default] ** 2024-01-23T08:45:37.815 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-23T08:45:37.815 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-23T08:45:37.815 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 2024-01-23T08:45:37.815 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 2024-01-23T08:45:37.815 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-23T08:45:37.815 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-01-23T08:45:37.815 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ** Compaction Stats [default] ** 2024-01-23T08:45:37.815 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-23T08:45:37.815 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-23T08:45:37.815 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 2024-01-23T08:45:37.815 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-23T08:45:37.815 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-01-23T08:45:37.816 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-23T08:45:37.816 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-01-23T08:45:37.816 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-01-23T08:45:37.816 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: AddFile(Keys): cumulative 0, interval 0 2024-01-23T08:45:37.816 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Cumulative compaction: 0.00 GB write, 0.41 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-23T08:45:37.816 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-23T08:45:37.816 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-23T08:45:37.816 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-01-23T08:45:37.816 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ** File Read Latency Histogram By Level [default] ** 2024-01-23T08:45:37.816 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: mon.smithi137 does not exist in monmap, will attempt to join an existing cluster 2024-01-23T08:45:37.816 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: using public_addr v2:172.21.15.137:0/0 -> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] 2024-01-23T08:45:37.816 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: starting mon.smithi137 rank -1 at public addrs [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] at bind addrs [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi137 fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 2024-01-23T08:45:37.816 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: mon.smithi137@-1(???) e0 preinit fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 2024-01-23T08:45:37.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: mon.smithi137@-1(synchronizing).mds e1 new map 2024-01-23T08:45:37.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: mon.smithi137@-1(synchronizing).mds e1 print_map 2024-01-23T08:45:37.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: e1 2024-01-23T08:45:37.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-01-23T08:45:37.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-23T08:45:37.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: legacy client fscid: -1 2024-01-23T08:45:37.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-01-23T08:45:37.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: No filesystems configured 2024-01-23T08:45:37.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: mon.smithi137@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-01-23T08:45:37.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: mon.smithi137@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-01-23T08:45:37.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: mon.smithi137@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-01-23T08:45:37.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: mon.smithi137@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-01-23T08:45:37.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: mon.smithi137@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-01-23T08:45:37.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: mon.smithi137@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-01-23T08:45:37.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: mon.smithi137@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-01-23T08:45:37.818 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: mon.smithi137@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-23T08:45:37.818 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: mon.smithi137@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-01-23T08:45:37.818 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: mon.smithi137@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-01-23T08:45:37.818 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: mon.smithi137@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-01-23T08:45:37.818 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:45:37.818 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.818 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.818 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.818 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.818 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.818 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: Deploying daemon node-exporter.smithi059 on smithi059 2024-01-23T08:45:37.818 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi137", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:45:37.818 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: Deploying cephadm binary to smithi137 2024-01-23T08:45:37.818 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.818 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.819 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.819 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: Added host smithi137 2024-01-23T08:45:37.819 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.819 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.819 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.819 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.819 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.819 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: Deploying daemon alertmanager.smithi059 on smithi059 2024-01-23T08:45:37.819 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.819 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.819 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-23T08:45:37.819 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.819 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.819 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/3557734549' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-23T08:45:37.820 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/3557734549' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-23T08:45:37.820 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: osdmap e4: 0 total, 0 up, 0 in 2024-01-23T08:45:37.820 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.820 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.820 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.820 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.820 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.820 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.820 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.820 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.820 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.820 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-23T08:45:37.820 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.821 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi059:172.21.15.59=smithi059;smithi137:172.21.15.137=smithi137", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:45:37.821 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: Saving service mon spec with placement smithi059:172.21.15.59=smithi059;smithi137:172.21.15.137=smithi137;count:2 2024-01-23T08:45:37.821 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-23T08:45:37.821 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: Deploying daemon grafana.smithi059 on smithi059 2024-01-23T08:45:37.821 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.821 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.821 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='client.? 172.21.15.137:0/3225837322' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:37.821 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.821 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.821 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='client.? 172.21.15.137:0/757064726' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:37.821 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.821 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.821 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='client.? 172.21.15.137:0/4245338094' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:37.821 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.821 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='client.? 172.21.15.137:0/4244508471' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:37.822 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.822 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.822 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.822 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.822 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.822 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.822 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.822 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.822 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: Deploying daemon prometheus.smithi059 on smithi059 2024-01-23T08:45:37.822 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='client.? 172.21.15.137:0/2467451709' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:37.822 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.822 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.822 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.822 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='client.? 172.21.15.137:0/1109412114' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:37.822 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.822 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.823 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='client.? 172.21.15.137:0/648622975' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:37.823 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.823 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.823 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.823 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.823 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-01-23T08:45:37.823 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14162 172.21.15.59:0/131470316' entity='mgr.smithi059.bzqirf' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-01-23T08:45:37.823 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: mgrmap e14: smithi059.bzqirf(active, since 59s) 2024-01-23T08:45:37.823 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='client.? 172.21.15.137:0/560299928' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:37.823 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='client.? 172.21.15.137:0/1544206647' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:37.823 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: Active manager daemon smithi059.bzqirf restarted 2024-01-23T08:45:37.823 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: Activating manager daemon smithi059.bzqirf 2024-01-23T08:45:37.823 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: osdmap e5: 0 total, 0 up, 0 in 2024-01-23T08:45:37.823 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: mgrmap e15: smithi059.bzqirf(active, starting, since 0.00306046s) 2024-01-23T08:45:37.823 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-01-23T08:45:37.824 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mgr metadata", "who": "smithi059.bzqirf", "id": "smithi059.bzqirf"}]: dispatch 2024-01-23T08:45:37.824 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-23T08:45:37.824 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-23T08:45:37.824 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-23T08:45:37.824 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: Manager daemon smithi059.bzqirf is now available 2024-01-23T08:45:37.824 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:45:37.824 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:45:37.824 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.824 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:45:37.824 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.bzqirf/mirror_snapshot_schedule"}]: dispatch 2024-01-23T08:45:37.824 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.bzqirf/trash_purge_schedule"}]: dispatch 2024-01-23T08:45:37.824 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.824 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='client.? 172.21.15.137:0/742870957' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:37.824 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: mgrmap e16: smithi059.bzqirf(active, since 1.00493s) 2024-01-23T08:45:37.825 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: [23/Jan/2024:08:45:11] ENGINE Bus STARTING 2024-01-23T08:45:37.825 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.825 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.825 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: [23/Jan/2024:08:45:11] ENGINE Serving on https://172.21.15.59:7150 2024-01-23T08:45:37.825 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: [23/Jan/2024:08:45:11] ENGINE Bus STARTED 2024-01-23T08:45:37.825 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.825 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: mgrmap e17: smithi059.bzqirf(active, since 2s) 2024-01-23T08:45:37.825 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.825 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.825 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.825 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.825 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd/host:smithi059", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:45:37.825 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.825 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.825 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.825 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd/host:smithi137", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:45:37.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:45:37.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:45:37.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: Updating smithi059:/etc/ceph/ceph.conf 2024-01-23T08:45:37.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: Updating smithi137:/etc/ceph/ceph.conf 2024-01-23T08:45:37.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: Updating smithi137:/var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-23T08:45:37.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: Updating smithi059:/var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-23T08:45:37.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-01-23T08:45:37.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='client.? 172.21.15.137:0/4045695746' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:37.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: Updating smithi059:/etc/ceph/ceph.client.admin.keyring 2024-01-23T08:45:37.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: Updating smithi137:/var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/config/ceph.client.admin.keyring 2024-01-23T08:45:37.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: Updating smithi059:/var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/config/ceph.client.admin.keyring 2024-01-23T08:45:37.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.827 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.827 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.827 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi137", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-23T08:45:37.827 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi137", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-23T08:45:37.827 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:45:37.827 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: Deploying daemon crash.smithi137 on smithi137 2024-01-23T08:45:37.827 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='client.? 172.21.15.137:0/1296824986' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:37.827 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.827 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.827 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.827 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.827 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: Deploying daemon node-exporter.smithi137 on smithi137 2024-01-23T08:45:37.827 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='client.? 172.21.15.137:0/520688025' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:37.827 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.827 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:45:37.828 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.828 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.828 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.828 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.828 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.ooirrh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-23T08:45:37.828 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.ooirrh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-01-23T08:45:37.828 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-23T08:45:37.828 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:45:37.828 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: Deploying daemon mgr.smithi137.ooirrh on smithi137 2024-01-23T08:45:37.828 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='client.? 172.21.15.137:0/205195856' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:37.828 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.828 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.828 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.828 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.828 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.829 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.829 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-23T08:45:37.829 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:45:37.829 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: Deploying daemon mon.smithi137 on smithi137 2024-01-23T08:45:37.829 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.829 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='client.? 172.21.15.137:0/1386400185' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:37.829 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.829 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:37.829 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:37.829 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: mon.smithi137@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-01-23T08:45:37.829 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 ceph-mon[35787]: mon.smithi137@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2024-01-23T08:45:37.829 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 bash[35735]: e1d4c945bf497c86dd42103b5aaa357d1975b38171b1e8a4dbfed858b29824e9 2024-01-23T08:45:37.829 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:37 smithi137 systemd[1]: Started Ceph mon.smithi137 for 3fc07a38-b9cb-11ee-95b1-87774f69a715. 2024-01-23T08:45:42.467 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:42.469+0000 7ff26d7fa700 1 -- 172.21.15.137:0/43023214 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1017 (secure 0 0 0) 0x7ff264011370 con 0x7ff270102ab0 2024-01-23T08:45:42.467 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-23T08:45:42.467 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":2,"fsid":"3fc07a38-b9cb-11ee-95b1-87774f69a715","modified":"2024-01-23T08:45:37.425255Z","created":"2024-01-23T08:43:01.494526Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi137","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:3300","nonce":0},{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-01-23T08:45:42.470 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:42.472+0000 7ff276a6b700 1 -- 172.21.15.137:0/43023214 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7ff2600367c0 msgr2=0x7ff260038c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:42.470 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:42.472+0000 7ff276a6b700 1 --2- 172.21.15.137:0/43023214 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7ff2600367c0 0x7ff260038c70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff25c009990 tx=0x7ff25c006e40 comp rx=0 tx=0).stop 2024-01-23T08:45:42.470 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:42.472+0000 7ff276a6b700 1 -- 172.21.15.137:0/43023214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff270102ab0 msgr2=0x7ff270112d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:42.470 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:42.472+0000 7ff276a6b700 1 --2- 172.21.15.137:0/43023214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff270102ab0 0x7ff270112d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff264009300 tx=0x7ff264009e30 comp rx=0 tx=0).stop 2024-01-23T08:45:42.471 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:42.472+0000 7ff276a6b700 1 -- 172.21.15.137:0/43023214 shutdown_connections 2024-01-23T08:45:42.471 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:42.472+0000 7ff276a6b700 1 --2- 172.21.15.137:0/43023214 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7ff2600367c0 0x7ff260038c70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:42.471 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:42.472+0000 7ff276a6b700 1 --2- 172.21.15.137:0/43023214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff270102ab0 0x7ff270112d40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:42.471 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:42.472+0000 7ff276a6b700 1 -- 172.21.15.137:0/43023214 >> 172.21.15.137:0/43023214 conn(0x7ff2700f85f0 msgr2=0x7ff2700fa580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:45:42.471 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:42.472+0000 7ff276a6b700 1 -- 172.21.15.137:0/43023214 shutdown_connections 2024-01-23T08:45:42.471 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:45:42.472+0000 7ff276a6b700 1 -- 172.21.15.137:0/43023214 wait complete. 2024-01-23T08:45:42.471 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 2 2024-01-23T08:45:42.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-01-23T08:45:42.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: mon.smithi059 calling monitor election 2024-01-23T08:45:42.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-01-23T08:45:42.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: from='mgr.? 172.21.15.137:0/2682570166' entity='mgr.smithi137.ooirrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.ooirrh/crt"}]: dispatch 2024-01-23T08:45:42.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: from='client.? 172.21.15.137:0/43023214' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:42.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:42.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-01-23T08:45:42.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-01-23T08:45:42.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: mon.smithi137 calling monitor election 2024-01-23T08:45:42.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:42.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:45:42.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-01-23T08:45:42.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-01-23T08:45:42.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-01-23T08:45:42.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: mon.smithi059 is new leader, mons smithi059,smithi137 in quorum (ranks 0,1) 2024-01-23T08:45:42.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: monmap e2: 2 mons at {smithi059=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0],smithi137=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-23T08:45:42.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: fsmap 2024-01-23T08:45:42.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: osdmap e5: 0 total, 0 up, 0 in 2024-01-23T08:45:42.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: mgrmap e17: smithi059.bzqirf(active, since 32s) 2024-01-23T08:45:42.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: Standby manager daemon smithi137.ooirrh started 2024-01-23T08:45:42.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: from='mgr.? 172.21.15.137:0/2682570166' entity='mgr.smithi137.ooirrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-23T08:45:42.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: overall HEALTH_OK 2024-01-23T08:45:42.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: from='mgr.? 172.21.15.137:0/2682570166' entity='mgr.smithi137.ooirrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.ooirrh/key"}]: dispatch 2024-01-23T08:45:42.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: from='mgr.? 172.21.15.137:0/2682570166' entity='mgr.smithi137.ooirrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-23T08:45:42.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:42.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:42.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:42.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:42.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:42 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:42.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-01-23T08:45:42.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: mon.smithi059 calling monitor election 2024-01-23T08:45:42.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-01-23T08:45:42.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: from='mgr.? 172.21.15.137:0/2682570166' entity='mgr.smithi137.ooirrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.ooirrh/crt"}]: dispatch 2024-01-23T08:45:42.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: from='client.? 172.21.15.137:0/43023214' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-23T08:45:42.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:42.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-01-23T08:45:42.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-01-23T08:45:42.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: mon.smithi137 calling monitor election 2024-01-23T08:45:42.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:42.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:45:42.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-01-23T08:45:42.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-01-23T08:45:42.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-01-23T08:45:42.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: mon.smithi059 is new leader, mons smithi059,smithi137 in quorum (ranks 0,1) 2024-01-23T08:45:42.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: monmap e2: 2 mons at {smithi059=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0],smithi137=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-23T08:45:42.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: fsmap 2024-01-23T08:45:42.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: osdmap e5: 0 total, 0 up, 0 in 2024-01-23T08:45:42.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: mgrmap e17: smithi059.bzqirf(active, since 32s) 2024-01-23T08:45:42.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: Standby manager daemon smithi137.ooirrh started 2024-01-23T08:45:42.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: from='mgr.? 172.21.15.137:0/2682570166' entity='mgr.smithi137.ooirrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-23T08:45:42.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: overall HEALTH_OK 2024-01-23T08:45:42.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: from='mgr.? 172.21.15.137:0/2682570166' entity='mgr.smithi137.ooirrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.ooirrh/key"}]: dispatch 2024-01-23T08:45:42.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: from='mgr.? 172.21.15.137:0/2682570166' entity='mgr.smithi137.ooirrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-23T08:45:42.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:42.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:42.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:42.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:42.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:42 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:43.120 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-01-23T08:45:43.121 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph config generate-minimal-conf 2024-01-23T08:45:43.389 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:45:43.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:43 smithi137 ceph-mon[35787]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:43.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:43 smithi137 ceph-mon[35787]: mgrmap e18: smithi059.bzqirf(active, since 32s), standbys: smithi137.ooirrh 2024-01-23T08:45:43.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:43 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mgr metadata", "who": "smithi137.ooirrh", "id": "smithi137.ooirrh"}]: dispatch 2024-01-23T08:45:43.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:43 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:45:43.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:43 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-01-23T08:45:43.808 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:43 smithi059 ceph-mon[28424]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:43.808 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:43 smithi059 ceph-mon[28424]: mgrmap e18: smithi059.bzqirf(active, since 32s), standbys: smithi137.ooirrh 2024-01-23T08:45:43.808 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:43 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mgr metadata", "who": "smithi137.ooirrh", "id": "smithi137.ooirrh"}]: dispatch 2024-01-23T08:45:43.809 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:43 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:45:43.809 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:43 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-01-23T08:45:44.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.937+0000 7ff9137cc700 1 -- 172.21.15.59:0/4188301614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff90c0fdac0 msgr2=0x7ff90c0fde90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:44.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.937+0000 7ff9137cc700 1 --2- 172.21.15.59:0/4188301614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff90c0fdac0 0x7ff90c0fde90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff9080048c0 tx=0x7ff908009e30 comp rx=0 tx=0).stop 2024-01-23T08:45:44.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.937+0000 7ff9137cc700 1 -- 172.21.15.59:0/4188301614 shutdown_connections 2024-01-23T08:45:44.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.937+0000 7ff9137cc700 1 --2- 172.21.15.59:0/4188301614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff90c0fdac0 0x7ff90c0fde90 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:44.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.937+0000 7ff9137cc700 1 -- 172.21.15.59:0/4188301614 >> 172.21.15.59:0/4188301614 conn(0x7ff90c0f7530 msgr2=0x7ff90c0f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:45:44.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.937+0000 7ff9137cc700 1 -- 172.21.15.59:0/4188301614 shutdown_connections 2024-01-23T08:45:44.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.937+0000 7ff9137cc700 1 -- 172.21.15.59:0/4188301614 wait complete. 2024-01-23T08:45:44.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.937+0000 7ff9137cc700 1 Processor -- start 2024-01-23T08:45:44.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.938+0000 7ff9137cc700 1 -- start start 2024-01-23T08:45:44.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.938+0000 7ff9137cc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff90c0fdac0 0x7ff90c112eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:44.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.938+0000 7ff9137cc700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff90c1133f0 0x7ff90c10cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:44.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.938+0000 7ff9137cc700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c113a00 con 0x7ff90c0fdac0 2024-01-23T08:45:44.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.938+0000 7ff9137cc700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c10d290 con 0x7ff90c1133f0 2024-01-23T08:45:44.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.938+0000 7ff911568700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff90c0fdac0 0x7ff90c112eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:45:44.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.938+0000 7ff911568700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff90c0fdac0 0x7ff90c112eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50408/0 (socket says 172.21.15.59:50408) 2024-01-23T08:45:44.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.938+0000 7ff911568700 1 -- 172.21.15.59:0/3176707970 learned_addr learned my addr 172.21.15.59:0/3176707970 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:45:44.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.938+0000 7ff910d67700 1 --2- 172.21.15.59:0/3176707970 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff90c1133f0 0x7ff90c10cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:45:44.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.938+0000 7ff910d67700 1 -- 172.21.15.59:0/3176707970 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff90c1133f0 msgr2=0x7ff90c10cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-01-23T08:45:44.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.938+0000 7ff910d67700 1 -- 172.21.15.59:0/3176707970 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff90c1133f0 msgr2=0x7ff90c10cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-01-23T08:45:44.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.938+0000 7ff910d67700 1 --2- 172.21.15.59:0/3176707970 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff90c1133f0 0x7ff90c10cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-01-23T08:45:44.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.938+0000 7ff910d67700 1 --2- 172.21.15.59:0/3176707970 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff90c1133f0 0x7ff90c10cd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-23T08:45:44.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.938+0000 7ff911568700 1 -- 172.21.15.59:0/3176707970 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff90c1133f0 msgr2=0x7ff90c10cd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-23T08:45:44.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.938+0000 7ff911568700 1 --2- 172.21.15.59:0/3176707970 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff90c1133f0 0x7ff90c10cd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:44.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.938+0000 7ff911568700 1 -- 172.21.15.59:0/3176707970 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff908005040 con 0x7ff90c0fdac0 2024-01-23T08:45:44.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.939+0000 7ff911568700 1 --2- 172.21.15.59:0/3176707970 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff90c0fdac0 0x7ff90c112eb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff90800bfe0 tx=0x7ff908009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:45:44.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.939+0000 7ff901ffb700 1 -- 172.21.15.59:0/3176707970 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff90800e040 con 0x7ff90c0fdac0 2024-01-23T08:45:44.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.939+0000 7ff901ffb700 1 -- 172.21.15.59:0/3176707970 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff908013420 con 0x7ff90c0fdac0 2024-01-23T08:45:44.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.939+0000 7ff9137cc700 1 -- 172.21.15.59:0/3176707970 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff90c10d510 con 0x7ff90c0fdac0 2024-01-23T08:45:44.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.939+0000 7ff901ffb700 1 -- 172.21.15.59:0/3176707970 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff90801c630 con 0x7ff90c0fdac0 2024-01-23T08:45:44.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.939+0000 7ff9137cc700 1 -- 172.21.15.59:0/3176707970 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff90c10da50 con 0x7ff90c0fdac0 2024-01-23T08:45:44.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.940+0000 7ff9137cc700 1 -- 172.21.15.59:0/3176707970 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff90c104260 con 0x7ff90c0fdac0 2024-01-23T08:45:44.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.940+0000 7ff901ffb700 1 -- 172.21.15.59:0/3176707970 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7ff908009580 con 0x7ff90c0fdac0 2024-01-23T08:45:44.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.941+0000 7ff901ffb700 1 --2- 172.21.15.59:0/3176707970 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7ff8f8069030 0x7ff8f806b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:44.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.941+0000 7ff901ffb700 1 -- 172.21.15.59:0/3176707970 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7ff908086f70 con 0x7ff90c0fdac0 2024-01-23T08:45:44.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.941+0000 7ff910d67700 1 --2- 172.21.15.59:0/3176707970 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7ff8f8069030 0x7ff8f806b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:45:44.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.941+0000 7ff910d67700 1 --2- 172.21.15.59:0/3176707970 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7ff8f8069030 0x7ff8f806b4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc006f30 tx=0x7ff8fc008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:45:44.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:44.945+0000 7ff901ffb700 1 -- 172.21.15.59:0/3176707970 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ff9080b8a00 con 0x7ff90c0fdac0 2024-01-23T08:45:45.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:45.145+0000 7ff9137cc700 1 -- 172.21.15.59:0/3176707970 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7ff90c05f450 con 0x7ff90c0fdac0 2024-01-23T08:45:45.144 INFO:teuthology.orchestra.run.smithi059.stdout:# minimal ceph.conf for 3fc07a38-b9cb-11ee-95b1-87774f69a715 2024-01-23T08:45:45.144 INFO:teuthology.orchestra.run.smithi059.stdout:[global] 2024-01-23T08:45:45.144 INFO:teuthology.orchestra.run.smithi059.stdout: fsid = 3fc07a38-b9cb-11ee-95b1-87774f69a715 2024-01-23T08:45:45.144 INFO:teuthology.orchestra.run.smithi059.stdout: mon_host = [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] 2024-01-23T08:45:45.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:45.145+0000 7ff901ffb700 1 -- 172.21.15.59:0/3176707970 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7ff908058050 con 0x7ff90c0fdac0 2024-01-23T08:45:45.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:45.148+0000 7ff9137cc700 1 -- 172.21.15.59:0/3176707970 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7ff8f8069030 msgr2=0x7ff8f806b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:45.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:45.148+0000 7ff9137cc700 1 --2- 172.21.15.59:0/3176707970 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7ff8f8069030 0x7ff8f806b4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc006f30 tx=0x7ff8fc008040 comp rx=0 tx=0).stop 2024-01-23T08:45:45.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:45.148+0000 7ff9137cc700 1 -- 172.21.15.59:0/3176707970 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff90c0fdac0 msgr2=0x7ff90c112eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:45.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:45.148+0000 7ff9137cc700 1 --2- 172.21.15.59:0/3176707970 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff90c0fdac0 0x7ff90c112eb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff90800bfe0 tx=0x7ff908009e30 comp rx=0 tx=0).stop 2024-01-23T08:45:45.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:45.148+0000 7ff9137cc700 1 -- 172.21.15.59:0/3176707970 shutdown_connections 2024-01-23T08:45:45.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:45.148+0000 7ff9137cc700 1 --2- 172.21.15.59:0/3176707970 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7ff8f8069030 0x7ff8f806b4e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:45.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:45.148+0000 7ff9137cc700 1 --2- 172.21.15.59:0/3176707970 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff90c0fdac0 0x7ff90c112eb0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:45.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:45.148+0000 7ff9137cc700 1 --2- 172.21.15.59:0/3176707970 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff90c1133f0 0x7ff90c10cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:45.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:45.148+0000 7ff9137cc700 1 -- 172.21.15.59:0/3176707970 >> 172.21.15.59:0/3176707970 conn(0x7ff90c0f7530 msgr2=0x7ff90c1013e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:45:45.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:45.148+0000 7ff9137cc700 1 -- 172.21.15.59:0/3176707970 shutdown_connections 2024-01-23T08:45:45.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:45.148+0000 7ff9137cc700 1 -- 172.21.15.59:0/3176707970 wait complete. 2024-01-23T08:45:45.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:45 smithi059 ceph-mon[28424]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:45.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:45 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:45.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:45 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:45.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:45 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:45:45.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:45 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:45:45.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:45 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/3176707970' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:45:45.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:45 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:45.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:45 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:45.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:45 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:45.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:45 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:45.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:45 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:45.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:45 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:45.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:45 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-23T08:45:45.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:45 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-23T08:45:45.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:45 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:45:45.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:45 smithi137 ceph-mon[35787]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:45.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:45 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:45.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:45 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:45.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:45 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:45:45.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:45 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:45:45.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:45 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/3176707970' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:45:45.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:45 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:45.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:45 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:45.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:45 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:45.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:45 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:45.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:45 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:45.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:45 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:45.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:45 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-23T08:45:45.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:45 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-23T08:45:45.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:45 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:45:45.847 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-01-23T08:45:45.848 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-23T08:45:45.848 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/ceph.conf 2024-01-23T08:45:45.879 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-23T08:45:45.879 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-23T08:45:45.946 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-23T08:45:45.946 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.conf 2024-01-23T08:45:45.979 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-23T08:45:45.979 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-23T08:45:46.048 INFO:tasks.cephadm:Deploying OSDs... 2024-01-23T08:45:46.049 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-23T08:45:46.049 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/scratch_devs of=/dev/stdout 2024-01-23T08:45:46.063 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-23T08:45:46.063 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_1 2024-01-23T08:45:46.118 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-23T08:45:46.118 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-23T08:45:46.118 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-01-23T08:45:46.118 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-23T08:45:46.118 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-01-23T08:45:46.118 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-01-23 08:45:01.935470239 +0000 2024-01-23T08:45:46.118 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-01-23 08:32:15.393744701 +0000 2024-01-23T08:45:46.118 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-01-23 08:32:15.393744701 +0000 2024-01-23T08:45:46.118 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-01-23 08:32:15.393744701 +0000 2024-01-23T08:45:46.119 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-23T08:45:46.183 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-01-23T08:45:46.184 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-01-23T08:45:46.184 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000261152 s, 2.0 MB/s 2024-01-23T08:45:46.185 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-23T08:45:46.241 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_2 2024-01-23T08:45:46.295 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-23T08:45:46.295 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-23T08:45:46.295 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-01-23T08:45:46.296 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-23T08:45:46.296 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-01-23T08:45:46.296 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-01-23 08:45:01.935470239 +0000 2024-01-23T08:45:46.296 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-01-23 08:32:15.743740116 +0000 2024-01-23T08:45:46.296 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-01-23 08:32:15.743740116 +0000 2024-01-23T08:45:46.296 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-01-23 08:32:15.743740116 +0000 2024-01-23T08:45:46.296 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-23T08:45:46.361 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-01-23T08:45:46.361 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-01-23T08:45:46.361 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000175569 s, 2.9 MB/s 2024-01-23T08:45:46.362 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-23T08:45:46.417 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_3 2024-01-23T08:45:46.471 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-23T08:45:46.472 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-23T08:45:46.472 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-01-23T08:45:46.472 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-23T08:45:46.472 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-01-23T08:45:46.472 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-01-23 08:45:01.936470226 +0000 2024-01-23T08:45:46.472 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-01-23 08:32:16.095735506 +0000 2024-01-23T08:45:46.472 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-01-23 08:32:16.095735506 +0000 2024-01-23T08:45:46.472 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-01-23 08:32:16.095735506 +0000 2024-01-23T08:45:46.472 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-23T08:45:46.537 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-01-23T08:45:46.537 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-01-23T08:45:46.537 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000185949 s, 2.8 MB/s 2024-01-23T08:45:46.539 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-23T08:45:46.594 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_4 2024-01-23T08:45:46.650 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-23T08:45:46.651 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-23T08:45:46.651 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-01-23T08:45:46.651 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-23T08:45:46.651 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-01-23T08:45:46.651 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-01-23 08:45:01.936470226 +0000 2024-01-23T08:45:46.651 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-01-23 08:32:16.450730856 +0000 2024-01-23T08:45:46.651 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-01-23 08:32:16.450730856 +0000 2024-01-23T08:45:46.651 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-01-23 08:32:16.450730856 +0000 2024-01-23T08:45:46.651 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-23T08:45:46.712 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:46 smithi059 ceph-mon[28424]: Updating smithi059:/etc/ceph/ceph.conf 2024-01-23T08:45:46.712 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:46 smithi059 ceph-mon[28424]: Updating smithi137:/etc/ceph/ceph.conf 2024-01-23T08:45:46.712 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:46 smithi059 ceph-mon[28424]: Updating smithi059:/var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-23T08:45:46.712 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:46 smithi059 ceph-mon[28424]: Updating smithi137:/var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-23T08:45:46.712 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:46 smithi059 ceph-mon[28424]: Reconfiguring mon.smithi059 (unknown last config time)... 2024-01-23T08:45:46.712 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:46 smithi059 ceph-mon[28424]: Reconfiguring daemon mon.smithi059 on smithi059 2024-01-23T08:45:46.717 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-01-23T08:45:46.717 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-01-23T08:45:46.717 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000236471 s, 2.2 MB/s 2024-01-23T08:45:46.718 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-23T08:45:46.773 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-23T08:45:46.773 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/scratch_devs of=/dev/stdout 2024-01-23T08:45:46.792 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-23T08:45:46.792 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_1 2024-01-23T08:45:46.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:46 smithi137 ceph-mon[35787]: Updating smithi059:/etc/ceph/ceph.conf 2024-01-23T08:45:46.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:46 smithi137 ceph-mon[35787]: Updating smithi137:/etc/ceph/ceph.conf 2024-01-23T08:45:46.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:46 smithi137 ceph-mon[35787]: Updating smithi059:/var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-23T08:45:46.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:46 smithi137 ceph-mon[35787]: Updating smithi137:/var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-23T08:45:46.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:46 smithi137 ceph-mon[35787]: Reconfiguring mon.smithi059 (unknown last config time)... 2024-01-23T08:45:46.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:46 smithi137 ceph-mon[35787]: Reconfiguring daemon mon.smithi059 on smithi059 2024-01-23T08:45:46.810 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-23T08:45:46.810 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-23T08:45:46.810 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-01-23T08:45:46.810 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-23T08:45:46.810 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-01-23T08:45:46.810 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-01-23 08:45:35.838949080 +0000 2024-01-23T08:45:46.810 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-01-23 08:32:15.293866715 +0000 2024-01-23T08:45:46.810 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-01-23 08:32:15.293866715 +0000 2024-01-23T08:45:46.810 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-01-23 08:32:15.293866715 +0000 2024-01-23T08:45:46.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-23T08:45:46.878 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-01-23T08:45:46.878 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-01-23T08:45:46.878 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000236529 s, 2.2 MB/s 2024-01-23T08:45:46.879 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-23T08:45:46.937 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_2 2024-01-23T08:45:46.994 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-23T08:45:46.994 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-23T08:45:46.994 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-01-23T08:45:46.994 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-23T08:45:46.994 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-01-23T08:45:46.994 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-01-23 08:45:35.838949080 +0000 2024-01-23T08:45:46.994 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-01-23 08:32:15.644859802 +0000 2024-01-23T08:45:46.994 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-01-23 08:32:15.644859802 +0000 2024-01-23T08:45:46.994 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-01-23 08:32:15.644859802 +0000 2024-01-23T08:45:46.995 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-23T08:45:47.062 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-01-23T08:45:47.062 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-01-23T08:45:47.062 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000191307 s, 2.7 MB/s 2024-01-23T08:45:47.064 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-23T08:45:47.120 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_3 2024-01-23T08:45:47.176 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-23T08:45:47.176 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-23T08:45:47.176 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-01-23T08:45:47.176 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-23T08:45:47.176 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-01-23T08:45:47.176 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-01-23 08:45:35.839949061 +0000 2024-01-23T08:45:47.176 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-01-23 08:32:15.988853025 +0000 2024-01-23T08:45:47.176 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-01-23 08:32:15.988853025 +0000 2024-01-23T08:45:47.177 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-01-23 08:32:15.988853025 +0000 2024-01-23T08:45:47.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-23T08:45:47.242 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-01-23T08:45:47.242 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-01-23T08:45:47.242 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000222161 s, 2.3 MB/s 2024-01-23T08:45:47.244 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-23T08:45:47.299 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_4 2024-01-23T08:45:47.354 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-23T08:45:47.354 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-23T08:45:47.354 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-01-23T08:45:47.354 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-23T08:45:47.355 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-01-23T08:45:47.355 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-01-23 08:45:35.839949061 +0000 2024-01-23T08:45:47.355 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-01-23 08:32:16.340846092 +0000 2024-01-23T08:45:47.355 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-01-23 08:32:16.340846092 +0000 2024-01-23T08:45:47.355 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-01-23 08:32:16.340846092 +0000 2024-01-23T08:45:47.355 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-23T08:45:47.420 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-01-23T08:45:47.420 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-01-23T08:45:47.420 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000215203 s, 2.4 MB/s 2024-01-23T08:45:47.421 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-23T08:45:47.477 INFO:tasks.cephadm:Deploying osd.0 on smithi059 with /dev/vg_nvme/lv_4... 2024-01-23T08:45:47.477 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-01-23T08:45:47.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:47 smithi059 ceph-mon[28424]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:47.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:47 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:47.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:47 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:47.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:47 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi059.bzqirf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-23T08:45:47.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:47 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-23T08:45:47.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:47 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:45:47.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:47 smithi137 ceph-mon[35787]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:47.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:47 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:47.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:47 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:47.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:47 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi059.bzqirf", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-23T08:45:47.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:47 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-23T08:45:47.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:47 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:45:48.399 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:45:48.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:48 smithi137 ceph-mon[35787]: Reconfiguring mgr.smithi059.bzqirf (unknown last config time)... 2024-01-23T08:45:48.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:48 smithi137 ceph-mon[35787]: Reconfiguring daemon mgr.smithi059.bzqirf on smithi059 2024-01-23T08:45:48.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:48 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:48.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:48 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:48.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:48 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi059", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-23T08:45:48.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:48 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:45:48.822 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:48 smithi059 ceph-mon[28424]: Reconfiguring mgr.smithi059.bzqirf (unknown last config time)... 2024-01-23T08:45:48.822 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:48 smithi059 ceph-mon[28424]: Reconfiguring daemon mgr.smithi059.bzqirf on smithi059 2024-01-23T08:45:48.822 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:48 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:48.822 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:48 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:48.822 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:48 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi059", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-23T08:45:48.822 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:48 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:45:49.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:49 smithi137 ceph-mon[35787]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:49.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:49 smithi137 ceph-mon[35787]: Reconfiguring crash.smithi059 (monmap changed)... 2024-01-23T08:45:49.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:49 smithi137 ceph-mon[35787]: Reconfiguring daemon crash.smithi059 on smithi059 2024-01-23T08:45:49.833 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:49 smithi059 ceph-mon[28424]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:49.833 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:49 smithi059 ceph-mon[28424]: Reconfiguring crash.smithi059 (monmap changed)... 2024-01-23T08:45:49.833 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:49 smithi059 ceph-mon[28424]: Reconfiguring daemon crash.smithi059 on smithi059 2024-01-23T08:45:51.004 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-01-23T08:45:51.004 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-01-23T08:45:51.004 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10+0 records in 2024-01-23T08:45:51.004 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2024-01-23T08:45:51.005 INFO:teuthology.orchestra.run.smithi059.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0137407 s, 763 MB/s 2024-01-23T08:45:51.005 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 2024-01-23T08:45:51.005 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping successful for: 2024-01-23T08:45:51.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:50 smithi137 ceph-mon[35787]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:51.320 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:50 smithi059 ceph-mon[28424]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:51.654 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi059:vg_nvme/lv_4 2024-01-23T08:45:51.926 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:45:52.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:52 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:52.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:52 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:52.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:52 smithi059 ceph-mon[28424]: Reconfiguring alertmanager.smithi059 (dependencies changed)... 2024-01-23T08:45:52.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:52 smithi059 ceph-mon[28424]: Reconfiguring daemon alertmanager.smithi059 on smithi059 2024-01-23T08:45:52.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:52 smithi059 ceph-mon[28424]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:53.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:52 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:53.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:52 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:53.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:52 smithi137 ceph-mon[35787]: Reconfiguring alertmanager.smithi059 (dependencies changed)... 2024-01-23T08:45:53.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:52 smithi137 ceph-mon[35787]: Reconfiguring daemon alertmanager.smithi059 on smithi059 2024-01-23T08:45:53.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:52 smithi137 ceph-mon[35787]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:54.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.319+0000 7f3d0456f700 1 -- 172.21.15.59:0/2127685746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cfc0fcee0 msgr2=0x7f3cfc0fd330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:54.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.319+0000 7f3d0456f700 1 --2- 172.21.15.59:0/2127685746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cfc0fcee0 0x7f3cfc0fd330 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3cec004940 tx=0x7f3cec009e30 comp rx=0 tx=0).stop 2024-01-23T08:45:54.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.319+0000 7f3d0456f700 1 -- 172.21.15.59:0/2127685746 shutdown_connections 2024-01-23T08:45:54.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.319+0000 7f3d0456f700 1 --2- 172.21.15.59:0/2127685746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cfc0fcee0 0x7f3cfc0fd330 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:54.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.319+0000 7f3d0456f700 1 --2- 172.21.15.59:0/2127685746 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cfc0fbcd0 0x7f3cfc0fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:54.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.319+0000 7f3d0456f700 1 -- 172.21.15.59:0/2127685746 >> 172.21.15.59:0/2127685746 conn(0x7f3cfc0f73b0 msgr2=0x7f3cfc0f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:45:54.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.319+0000 7f3d0456f700 1 -- 172.21.15.59:0/2127685746 shutdown_connections 2024-01-23T08:45:54.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.319+0000 7f3d0456f700 1 -- 172.21.15.59:0/2127685746 wait complete. 2024-01-23T08:45:54.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.320+0000 7f3d0456f700 1 Processor -- start 2024-01-23T08:45:54.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.320+0000 7f3d0456f700 1 -- start start 2024-01-23T08:45:54.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.320+0000 7f3d0456f700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cfc0fbcd0 0x7f3cfc0ffda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:54.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.320+0000 7f3d0456f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cfc0fcee0 0x7f3cfc0fe220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:54.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.320+0000 7f3d0456f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc0fe8b0 con 0x7f3cfc0fcee0 2024-01-23T08:45:54.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.320+0000 7f3d0456f700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc0fea20 con 0x7f3cfc0fbcd0 2024-01-23T08:45:54.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.320+0000 7f3d01b0a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cfc0fcee0 0x7f3cfc0fe220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:45:54.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.320+0000 7f3d01b0a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cfc0fcee0 0x7f3cfc0fe220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51148/0 (socket says 172.21.15.59:51148) 2024-01-23T08:45:54.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.320+0000 7f3d01b0a700 1 -- 172.21.15.59:0/123770489 learned_addr learned my addr 172.21.15.59:0/123770489 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:45:54.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.320+0000 7f3d0230b700 1 --2- 172.21.15.59:0/123770489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cfc0fbcd0 0x7f3cfc0ffda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:45:54.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.320+0000 7f3d01b0a700 1 -- 172.21.15.59:0/123770489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cfc0fbcd0 msgr2=0x7f3cfc0ffda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:45:54.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.320+0000 7f3d01b0a700 1 --2- 172.21.15.59:0/123770489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cfc0fbcd0 0x7f3cfc0ffda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:45:54.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.320+0000 7f3d01b0a700 1 -- 172.21.15.59:0/123770489 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cec005040 con 0x7f3cfc0fcee0 2024-01-23T08:45:54.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.321+0000 7f3d0230b700 1 --2- 172.21.15.59:0/123770489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cfc0fbcd0 0x7f3cfc0ffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-23T08:45:54.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.321+0000 7f3d01b0a700 1 --2- 172.21.15.59:0/123770489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cfc0fcee0 0x7f3cfc0fe220 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3cec009360 tx=0x7f3cec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:45:54.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.321+0000 7f3cf2ffd700 1 -- 172.21.15.59:0/123770489 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cec00e040 con 0x7f3cfc0fcee0 2024-01-23T08:45:54.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.321+0000 7f3cf2ffd700 1 -- 172.21.15.59:0/123770489 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3cec00bcd0 con 0x7f3cfc0fcee0 2024-01-23T08:45:54.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.321+0000 7f3d0456f700 1 -- 172.21.15.59:0/123770489 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cfc0feca0 con 0x7f3cfc0fcee0 2024-01-23T08:45:54.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.321+0000 7f3cf2ffd700 1 -- 172.21.15.59:0/123770489 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cec01c630 con 0x7f3cfc0fcee0 2024-01-23T08:45:54.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.321+0000 7f3d0456f700 1 -- 172.21.15.59:0/123770489 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cfc115750 con 0x7f3cfc0fcee0 2024-01-23T08:45:54.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.322+0000 7f3d0456f700 1 -- 172.21.15.59:0/123770489 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cfc04bd60 con 0x7f3cfc0fcee0 2024-01-23T08:45:54.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.323+0000 7f3cf2ffd700 1 -- 172.21.15.59:0/123770489 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7f3cec013960 con 0x7f3cfc0fcee0 2024-01-23T08:45:54.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.323+0000 7f3cf2ffd700 1 --2- 172.21.15.59:0/123770489 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f3ce8068ed0 0x7f3ce806b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:45:54.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.323+0000 7f3cf2ffd700 1 -- 172.21.15.59:0/123770489 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f3cec087000 con 0x7f3cfc0fcee0 2024-01-23T08:45:54.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.323+0000 7f3d0230b700 1 --2- 172.21.15.59:0/123770489 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f3ce8068ed0 0x7f3ce806b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:45:54.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.323+0000 7f3d0230b700 1 --2- 172.21.15.59:0/123770489 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f3ce8068ed0 0x7f3ce806b380 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8006fd0 tx=0x7f3cf8008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:45:54.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.328+0000 7f3cf2ffd700 1 -- 172.21.15.59:0/123770489 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3cec058190 con 0x7f3cfc0fcee0 2024-01-23T08:45:54.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:45:54.533+0000 7f3d0456f700 1 -- 172.21.15.59:0/123770489 --> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f3cfc0664f0 con 0x7f3ce8068ed0 2024-01-23T08:45:55.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:55 smithi059 ceph-mon[28424]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:55.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:55 smithi059 ceph-mon[28424]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:45:55.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:55 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-23T08:45:55.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:55 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-23T08:45:55.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:55 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:45:55.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:55 smithi137 ceph-mon[35787]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:55.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:55 smithi137 ceph-mon[35787]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:45:55.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:55 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-23T08:45:55.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:55 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-23T08:45:55.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:55 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:45:56.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:56 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:45:56.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:56 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:45:57.706 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:57 smithi059 ceph-mon[28424]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:57.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:57 smithi137 ceph-mon[35787]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:59.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:59 smithi059 ceph-mon[28424]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:59.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:59 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:59.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:45:59 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:59.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:59 smithi137 ceph-mon[35787]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:45:59.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:59 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:45:59.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:45:59 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:00.623 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:00 smithi059 ceph-mon[28424]: Reconfiguring grafana.smithi059 (dependencies changed)... 2024-01-23T08:46:00.623 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:00 smithi059 ceph-mon[28424]: Reconfiguring daemon grafana.smithi059 on smithi059 2024-01-23T08:46:00.624 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:00 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/3734728467' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "444a9c2f-e580-443e-9dfe-36843f8c2e13"}]: dispatch 2024-01-23T08:46:00.624 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:00 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/3734728467' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "444a9c2f-e580-443e-9dfe-36843f8c2e13"}]': finished 2024-01-23T08:46:00.624 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:00 smithi059 ceph-mon[28424]: osdmap e6: 1 total, 0 up, 1 in 2024-01-23T08:46:00.624 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:00 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-23T08:46:00.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:00 smithi137 ceph-mon[35787]: Reconfiguring grafana.smithi059 (dependencies changed)... 2024-01-23T08:46:00.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:00 smithi137 ceph-mon[35787]: Reconfiguring daemon grafana.smithi059 on smithi059 2024-01-23T08:46:00.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:00 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/3734728467' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "444a9c2f-e580-443e-9dfe-36843f8c2e13"}]: dispatch 2024-01-23T08:46:00.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:00 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/3734728467' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "444a9c2f-e580-443e-9dfe-36843f8c2e13"}]': finished 2024-01-23T08:46:00.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:00 smithi137 ceph-mon[35787]: osdmap e6: 1 total, 0 up, 1 in 2024-01-23T08:46:00.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:00 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-23T08:46:01.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:01 smithi059 ceph-mon[28424]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:01.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:01 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/4291079959' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-23T08:46:01.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:01 smithi137 ceph-mon[35787]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:01.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:01 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/4291079959' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-23T08:46:03.679 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:03 smithi059 ceph-mon[28424]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:03.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:03 smithi137 ceph-mon[35787]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:05.566 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:05 smithi059 ceph-mon[28424]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:05.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:05 smithi137 ceph-mon[35787]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:07.688 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:07 smithi059 ceph-mon[28424]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:07.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:07 smithi137 ceph-mon[35787]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:09.521 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:09 smithi059 ceph-mon[28424]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:09.522 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:09 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:09.522 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:09 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:09.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:09 smithi137 ceph-mon[35787]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:09.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:09 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:09.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:09 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:10.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:10 smithi059 ceph-mon[28424]: Reconfiguring prometheus.smithi059 (dependencies changed)... 2024-01-23T08:46:10.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:10 smithi059 ceph-mon[28424]: Reconfiguring daemon prometheus.smithi059 on smithi059 2024-01-23T08:46:10.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:10 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:46:10.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:10 smithi137 ceph-mon[35787]: Reconfiguring prometheus.smithi059 (dependencies changed)... 2024-01-23T08:46:10.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:10 smithi137 ceph-mon[35787]: Reconfiguring daemon prometheus.smithi059 on smithi059 2024-01-23T08:46:10.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:10 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:46:11.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:11 smithi059 ceph-mon[28424]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:11.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:11 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-01-23T08:46:11.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:11 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:46:11.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:11 smithi137 ceph-mon[35787]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:11.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:11 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-01-23T08:46:11.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:11 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:46:12.656 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:12 smithi059 ceph-mon[28424]: Deploying daemon osd.0 on smithi059 2024-01-23T08:46:12.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:12 smithi137 ceph-mon[35787]: Deploying daemon osd.0 on smithi059 2024-01-23T08:46:13.694 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:13 smithi137 ceph-mon[35787]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:13.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:13 smithi059 ceph-mon[28424]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:15.665 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:15 smithi137 ceph-mon[35787]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:15.665 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:15 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:15.665 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:15 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:15.665 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:15 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi137", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-23T08:46:15.665 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:15 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:46:15.681 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:15 smithi059 ceph-mon[28424]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:15.681 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:15 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:15.681 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:15 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:15.681 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:15 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi137", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-23T08:46:15.682 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:15 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:46:16.499 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:16 smithi137 ceph-mon[35787]: Reconfiguring crash.smithi137 (monmap changed)... 2024-01-23T08:46:16.499 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:16 smithi137 ceph-mon[35787]: Reconfiguring daemon crash.smithi137 on smithi137 2024-01-23T08:46:16.619 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:16 smithi059 ceph-mon[28424]: Reconfiguring crash.smithi137 (monmap changed)... 2024-01-23T08:46:16.619 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:16 smithi059 ceph-mon[28424]: Reconfiguring daemon crash.smithi137 on smithi137 2024-01-23T08:46:17.865 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:17 smithi059 ceph-mon[28424]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:17.865 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:17 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:17.865 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:17 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:17.865 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:17 smithi059 ceph-mon[28424]: Reconfiguring mgr.smithi137.ooirrh (monmap changed)... 2024-01-23T08:46:17.865 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:17 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.ooirrh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-23T08:46:17.866 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:17 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-23T08:46:17.866 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:17 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:46:17.866 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:17 smithi059 ceph-mon[28424]: Reconfiguring daemon mgr.smithi137.ooirrh on smithi137 2024-01-23T08:46:17.919 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:17 smithi137 ceph-mon[35787]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:17.919 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:17 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:17.920 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:17 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:17.920 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:17 smithi137 ceph-mon[35787]: Reconfiguring mgr.smithi137.ooirrh (monmap changed)... 2024-01-23T08:46:17.920 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:17 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.ooirrh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-23T08:46:17.920 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:17 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-23T08:46:17.920 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:17 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:46:17.920 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:17 smithi137 ceph-mon[35787]: Reconfiguring daemon mgr.smithi137.ooirrh on smithi137 2024-01-23T08:46:19.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:18 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:19.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:18 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:19.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:18 smithi137 ceph-mon[35787]: Reconfiguring mon.smithi137 (monmap changed)... 2024-01-23T08:46:19.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:18 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-23T08:46:19.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:18 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-23T08:46:19.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:18 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:46:19.044 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:18 smithi137 ceph-mon[35787]: Reconfiguring daemon mon.smithi137 on smithi137 2024-01-23T08:46:19.044 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:18 smithi137 ceph-mon[35787]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:19.053 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:18 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:19.053 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:18 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:19.053 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:18 smithi059 ceph-mon[28424]: Reconfiguring mon.smithi137 (monmap changed)... 2024-01-23T08:46:19.053 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:18 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-23T08:46:19.053 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:18 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-23T08:46:19.053 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:18 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:46:19.053 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:18 smithi059 ceph-mon[28424]: Reconfiguring daemon mon.smithi137 on smithi137 2024-01-23T08:46:19.053 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:18 smithi059 ceph-mon[28424]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:20.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:20 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:20.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:20 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:20.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:20 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-23T08:46:20.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:20 smithi137 ceph-mon[35787]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-23T08:46:20.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:20 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi059.front.sepia.ceph.com:9093"}]: dispatch 2024-01-23T08:46:20.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:20 smithi137 ceph-mon[35787]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi059.front.sepia.ceph.com:9093"}]: dispatch 2024-01-23T08:46:20.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:20 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:20.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:20 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-23T08:46:20.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:20 smithi137 ceph-mon[35787]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-23T08:46:20.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:20 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi059.front.sepia.ceph.com:3000"}]: dispatch 2024-01-23T08:46:20.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:20 smithi137 ceph-mon[35787]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi059.front.sepia.ceph.com:3000"}]: dispatch 2024-01-23T08:46:20.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:20 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:20.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:20 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-23T08:46:20.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:20 smithi137 ceph-mon[35787]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-23T08:46:20.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:20 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi059.front.sepia.ceph.com:9095"}]: dispatch 2024-01-23T08:46:20.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:20 smithi137 ceph-mon[35787]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi059.front.sepia.ceph.com:9095"}]: dispatch 2024-01-23T08:46:20.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:20 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:20.295 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:20 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:46:20.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:20 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:20.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:20 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:20.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:20 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-23T08:46:20.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:20 smithi059 ceph-mon[28424]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-23T08:46:20.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:20 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi059.front.sepia.ceph.com:9093"}]: dispatch 2024-01-23T08:46:20.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:20 smithi059 ceph-mon[28424]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi059.front.sepia.ceph.com:9093"}]: dispatch 2024-01-23T08:46:20.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:20 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:20.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:20 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-23T08:46:20.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:20 smithi059 ceph-mon[28424]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-23T08:46:20.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:20 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi059.front.sepia.ceph.com:3000"}]: dispatch 2024-01-23T08:46:20.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:20 smithi059 ceph-mon[28424]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi059.front.sepia.ceph.com:3000"}]: dispatch 2024-01-23T08:46:20.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:20 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:20.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:20 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-23T08:46:20.473 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:20 smithi059 ceph-mon[28424]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-23T08:46:20.473 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:20 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi059.front.sepia.ceph.com:9095"}]: dispatch 2024-01-23T08:46:20.473 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:20 smithi059 ceph-mon[28424]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi059.front.sepia.ceph.com:9095"}]: dispatch 2024-01-23T08:46:20.473 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:20 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:20.473 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:20 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:46:21.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:21 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:21.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:21 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:21.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:21 smithi059 ceph-mon[28424]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:21.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:21 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:21.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:21 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:21.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:21 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:21.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:21 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:21.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:21 smithi137 ceph-mon[35787]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:21.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:21 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:21.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:21 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:23.705 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:23 smithi059 ceph-mon[28424]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:23.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:23 smithi137 ceph-mon[35787]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:24.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:24.281+0000 7f3cf2ffd700 1 -- 172.21.15.59:0/123770489 <== mgr.14210 v2:172.21.15.59:6800/3219735162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3cfc0664f0 con 0x7f3ce8068ed0 2024-01-23T08:46:24.279 INFO:teuthology.orchestra.run.smithi059.stdout:Created osd(s) 0 on host 'smithi059' 2024-01-23T08:46:24.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:24.283+0000 7f3d0456f700 1 -- 172.21.15.59:0/123770489 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f3ce8068ed0 msgr2=0x7f3ce806b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:46:24.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:24.283+0000 7f3d0456f700 1 --2- 172.21.15.59:0/123770489 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f3ce8068ed0 0x7f3ce806b380 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8006fd0 tx=0x7f3cf8008040 comp rx=0 tx=0).stop 2024-01-23T08:46:24.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:24.283+0000 7f3d0456f700 1 -- 172.21.15.59:0/123770489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cfc0fcee0 msgr2=0x7f3cfc0fe220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:46:24.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:24.283+0000 7f3d0456f700 1 --2- 172.21.15.59:0/123770489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cfc0fcee0 0x7f3cfc0fe220 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3cec009360 tx=0x7f3cec009e30 comp rx=0 tx=0).stop 2024-01-23T08:46:24.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:24.283+0000 7f3d0456f700 1 -- 172.21.15.59:0/123770489 shutdown_connections 2024-01-23T08:46:24.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:24.283+0000 7f3d0456f700 1 --2- 172.21.15.59:0/123770489 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f3ce8068ed0 0x7f3ce806b380 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:46:24.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:24.283+0000 7f3d0456f700 1 --2- 172.21.15.59:0/123770489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cfc0fbcd0 0x7f3cfc0ffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:46:24.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:24.283+0000 7f3d0456f700 1 --2- 172.21.15.59:0/123770489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3cfc0fcee0 0x7f3cfc0fe220 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:46:24.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:24.283+0000 7f3d0456f700 1 -- 172.21.15.59:0/123770489 >> 172.21.15.59:0/123770489 conn(0x7f3cfc0f73b0 msgr2=0x7f3cfc065de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:46:24.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:24.283+0000 7f3d0456f700 1 -- 172.21.15.59:0/123770489 shutdown_connections 2024-01-23T08:46:24.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:24.283+0000 7f3d0456f700 1 -- 172.21.15.59:0/123770489 wait complete. 2024-01-23T08:46:24.578 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:24 smithi059 ceph-mon[28424]: from='osd.0 [v2:172.21.15.59:6802/2564662034,v1:172.21.15.59:6803/2564662034]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-01-23T08:46:24.578 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:24 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:24.578 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:24 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:24.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:24 smithi137 ceph-mon[35787]: from='osd.0 [v2:172.21.15.59:6802/2564662034,v1:172.21.15.59:6803/2564662034]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-01-23T08:46:24.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:24 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:24.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:24 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:24.888 DEBUG:teuthology.orchestra.run.smithi059:osd.0> sudo journalctl -f -n 0 -u ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@osd.0.service 2024-01-23T08:46:24.890 INFO:tasks.cephadm:Deploying osd.1 on smithi059 with /dev/vg_nvme/lv_3... 2024-01-23T08:46:24.891 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-01-23T08:46:25.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:25 smithi059 ceph-mon[28424]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:25.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:25 smithi059 ceph-mon[28424]: from='osd.0 [v2:172.21.15.59:6802/2564662034,v1:172.21.15.59:6803/2564662034]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-01-23T08:46:25.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:25 smithi059 ceph-mon[28424]: osdmap e7: 1 total, 0 up, 1 in 2024-01-23T08:46:25.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:25 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-23T08:46:25.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:25 smithi059 ceph-mon[28424]: from='osd.0 [v2:172.21.15.59:6802/2564662034,v1:172.21.15.59:6803/2564662034]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-01-23T08:46:25.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:25 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:46:25.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:46:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:46:25.388+0000 7f707cd35700 -1 osd.0 0 waiting for initial osdmap 2024-01-23T08:46:25.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:46:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:46:25.394+0000 7f7077b30700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-23T08:46:25.693 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:46:25.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:25 smithi137 ceph-mon[35787]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:25.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:25 smithi137 ceph-mon[35787]: from='osd.0 [v2:172.21.15.59:6802/2564662034,v1:172.21.15.59:6803/2564662034]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-01-23T08:46:25.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:25 smithi137 ceph-mon[35787]: osdmap e7: 1 total, 0 up, 1 in 2024-01-23T08:46:25.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:25 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-23T08:46:25.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:25 smithi137 ceph-mon[35787]: from='osd.0 [v2:172.21.15.59:6802/2564662034,v1:172.21.15.59:6803/2564662034]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-01-23T08:46:25.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:25 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:46:26.705 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:26 smithi059 ceph-mon[28424]: from='osd.0 [v2:172.21.15.59:6802/2564662034,v1:172.21.15.59:6803/2564662034]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2024-01-23T08:46:26.706 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:26 smithi059 ceph-mon[28424]: osdmap e8: 1 total, 0 up, 1 in 2024-01-23T08:46:26.706 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:26 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-23T08:46:26.706 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:26 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-23T08:46:26.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:26 smithi137 ceph-mon[35787]: from='osd.0 [v2:172.21.15.59:6802/2564662034,v1:172.21.15.59:6803/2564662034]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2024-01-23T08:46:26.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:26 smithi137 ceph-mon[35787]: osdmap e8: 1 total, 0 up, 1 in 2024-01-23T08:46:26.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:26 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-23T08:46:26.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:26 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-23T08:46:27.594 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:27 smithi059 ceph-mon[28424]: purged_snaps scrub starts 2024-01-23T08:46:27.594 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:27 smithi059 ceph-mon[28424]: purged_snaps scrub ok 2024-01-23T08:46:27.594 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:27 smithi059 ceph-mon[28424]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:27.594 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:27 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-23T08:46:27.595 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:27 smithi059 ceph-mon[28424]: osd.0 [v2:172.21.15.59:6802/2564662034,v1:172.21.15.59:6803/2564662034] boot 2024-01-23T08:46:27.595 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:27 smithi059 ceph-mon[28424]: osdmap e9: 1 total, 1 up, 1 in 2024-01-23T08:46:27.595 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:27 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-23T08:46:27.595 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:27 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:27.595 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:27 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:27.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:27 smithi137 ceph-mon[35787]: purged_snaps scrub starts 2024-01-23T08:46:27.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:27 smithi137 ceph-mon[35787]: purged_snaps scrub ok 2024-01-23T08:46:27.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:27 smithi137 ceph-mon[35787]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-23T08:46:27.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:27 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-23T08:46:27.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:27 smithi137 ceph-mon[35787]: osd.0 [v2:172.21.15.59:6802/2564662034,v1:172.21.15.59:6803/2564662034] boot 2024-01-23T08:46:27.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:27 smithi137 ceph-mon[35787]: osdmap e9: 1 total, 1 up, 1 in 2024-01-23T08:46:27.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:27 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-23T08:46:27.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:27 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:27.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:27 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:28.104 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-01-23T08:46:28.104 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-01-23T08:46:28.105 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10+0 records in 2024-01-23T08:46:28.105 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2024-01-23T08:46:28.105 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125541 s, 835 MB/s 2024-01-23T08:46:28.105 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping successful for: 2024-01-23T08:46:28.853 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:28 smithi059 ceph-mon[28424]: osdmap e10: 1 total, 1 up, 1 in 2024-01-23T08:46:29.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:28 smithi137 ceph-mon[35787]: osdmap e10: 1 total, 1 up, 1 in 2024-01-23T08:46:29.178 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi059:vg_nvme/lv_3 2024-01-23T08:46:29.622 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:46:29.819 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:29 smithi059 ceph-mon[28424]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:30.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:29 smithi137 ceph-mon[35787]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:31.217 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:31 smithi059 ceph-mon[28424]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:31.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:31 smithi137 ceph-mon[35787]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:32.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.220+0000 7f1bac228700 1 -- 172.21.15.59:0/157182262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ba40fe490 msgr2=0x7f1ba40fe8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:46:32.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.220+0000 7f1bac228700 1 --2- 172.21.15.59:0/157182262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ba40fe490 0x7f1ba40fe8a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0004880 tx=0x7f1ba0009e30 comp rx=0 tx=0).stop 2024-01-23T08:46:32.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.221+0000 7f1bac228700 1 -- 172.21.15.59:0/157182262 shutdown_connections 2024-01-23T08:46:32.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.221+0000 7f1bac228700 1 --2- 172.21.15.59:0/157182262 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ba40fede0 0x7f1ba40faf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:46:32.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.221+0000 7f1bac228700 1 --2- 172.21.15.59:0/157182262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ba40fe490 0x7f1ba40fe8a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:46:32.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.221+0000 7f1bac228700 1 -- 172.21.15.59:0/157182262 >> 172.21.15.59:0/157182262 conn(0x7f1ba40f6bb0 msgr2=0x7f1ba40f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:46:32.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.221+0000 7f1bac228700 1 -- 172.21.15.59:0/157182262 shutdown_connections 2024-01-23T08:46:32.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.221+0000 7f1bac228700 1 -- 172.21.15.59:0/157182262 wait complete. 2024-01-23T08:46:32.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.221+0000 7f1bac228700 1 Processor -- start 2024-01-23T08:46:32.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.221+0000 7f1bac228700 1 -- start start 2024-01-23T08:46:32.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.221+0000 7f1bac228700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ba40fe490 0x7f1ba40fdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:46:32.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.221+0000 7f1bac228700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ba40fede0 0x7f1ba40fc150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:46:32.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.221+0000 7f1bac228700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba40fc750 con 0x7f1ba40fe490 2024-01-23T08:46:32.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.221+0000 7f1bac228700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba40fc8c0 con 0x7f1ba40fede0 2024-01-23T08:46:32.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.221+0000 7f1ba9fc4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ba40fe490 0x7f1ba40fdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:46:32.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.221+0000 7f1ba9fc4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ba40fe490 0x7f1ba40fdcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40646/0 (socket says 172.21.15.59:40646) 2024-01-23T08:46:32.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.221+0000 7f1ba9fc4700 1 -- 172.21.15.59:0/4174337808 learned_addr learned my addr 172.21.15.59:0/4174337808 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:46:32.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.222+0000 7f1ba97c3700 1 --2- 172.21.15.59:0/4174337808 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ba40fede0 0x7f1ba40fc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:46:32.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.222+0000 7f1ba9fc4700 1 -- 172.21.15.59:0/4174337808 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ba40fede0 msgr2=0x7f1ba40fc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:46:32.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.222+0000 7f1ba9fc4700 1 --2- 172.21.15.59:0/4174337808 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ba40fede0 0x7f1ba40fc150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:46:32.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.222+0000 7f1ba9fc4700 1 -- 172.21.15.59:0/4174337808 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba0005040 con 0x7f1ba40fe490 2024-01-23T08:46:32.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.222+0000 7f1ba97c3700 1 --2- 172.21.15.59:0/4174337808 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ba40fede0 0x7f1ba40fc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-23T08:46:32.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.222+0000 7f1ba9fc4700 1 --2- 172.21.15.59:0/4174337808 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ba40fe490 0x7f1ba40fdcd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1ba00134a0 tx=0x7f1ba0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:46:32.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.222+0000 7f1b967fc700 1 -- 172.21.15.59:0/4174337808 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ba000e030 con 0x7f1ba40fe490 2024-01-23T08:46:32.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.222+0000 7f1b967fc700 1 -- 172.21.15.59:0/4174337808 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ba00093c0 con 0x7f1ba40fe490 2024-01-23T08:46:32.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.222+0000 7f1bac228700 1 -- 172.21.15.59:0/4174337808 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba40fcb40 con 0x7f1ba40fe490 2024-01-23T08:46:32.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.222+0000 7f1b967fc700 1 -- 172.21.15.59:0/4174337808 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ba00123c0 con 0x7f1ba40fe490 2024-01-23T08:46:32.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.222+0000 7f1bac228700 1 -- 172.21.15.59:0/4174337808 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba4115ab0 con 0x7f1ba40fe490 2024-01-23T08:46:32.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.223+0000 7f1bac228700 1 -- 172.21.15.59:0/4174337808 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba41076f0 con 0x7f1ba40fe490 2024-01-23T08:46:32.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.223+0000 7f1b967fc700 1 -- 172.21.15.59:0/4174337808 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7f1ba0013c30 con 0x7f1ba40fe490 2024-01-23T08:46:32.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.224+0000 7f1b967fc700 1 --2- 172.21.15.59:0/4174337808 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f1b90068ed0 0x7f1b9006b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:46:32.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.224+0000 7f1b967fc700 1 -- 172.21.15.59:0/4174337808 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7f1ba00879d0 con 0x7f1ba40fe490 2024-01-23T08:46:32.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.224+0000 7f1ba97c3700 1 --2- 172.21.15.59:0/4174337808 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f1b90068ed0 0x7f1b9006b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:46:32.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.224+0000 7f1ba97c3700 1 --2- 172.21.15.59:0/4174337808 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f1b90068ed0 0x7f1b9006b380 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1b98007fd0 tx=0x7f1b9800b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:46:32.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.229+0000 7f1b967fc700 1 -- 172.21.15.59:0/4174337808 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1ba00587b0 con 0x7f1ba40fe490 2024-01-23T08:46:32.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:32.436+0000 7f1bac228700 1 -- 172.21.15.59:0/4174337808 --> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ba4105780 con 0x7f1b90068ed0 2024-01-23T08:46:32.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:32 smithi137 ceph-mon[35787]: Detected new or changed devices on smithi059 2024-01-23T08:46:32.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:32 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:32.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:32 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:32.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:32 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:46:32.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:32 smithi137 ceph-mon[35787]: Adjusting osd_memory_target on smithi059 to 12869M 2024-01-23T08:46:32.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:32 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:32.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:32 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:46:32.544 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:32 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:46:32.544 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:32 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:32.544 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:32 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:46:32.553 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:32 smithi059 ceph-mon[28424]: Detected new or changed devices on smithi059 2024-01-23T08:46:32.553 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:32 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:32.553 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:32 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:32.553 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:32 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:46:32.553 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:32 smithi059 ceph-mon[28424]: Adjusting osd_memory_target on smithi059 to 12869M 2024-01-23T08:46:32.553 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:32 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:32.553 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:32 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:46:32.553 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:32 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:46:32.553 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:32 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:32.553 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:32 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:46:33.313 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:33 smithi059 ceph-mon[28424]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:33.313 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:33 smithi059 ceph-mon[28424]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:46:33.313 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:33 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-23T08:46:33.313 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:33 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-23T08:46:33.313 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:33 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:46:33.313 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:33 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:33.313 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:33 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:33.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:33 smithi137 ceph-mon[35787]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:33.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:33 smithi137 ceph-mon[35787]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:46:33.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:33 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-23T08:46:33.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:33 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-23T08:46:33.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:33 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:46:33.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:33 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:33.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:33 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:35.668 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:35 smithi059 ceph-mon[28424]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:35.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:35 smithi137 ceph-mon[35787]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:36.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:36 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/120744383' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c28ec1b9-2f5f-48e6-8322-7092bbe8cdda"}]: dispatch 2024-01-23T08:46:36.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:36 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/120744383' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c28ec1b9-2f5f-48e6-8322-7092bbe8cdda"}]': finished 2024-01-23T08:46:36.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:36 smithi059 ceph-mon[28424]: osdmap e11: 2 total, 1 up, 2 in 2024-01-23T08:46:36.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:36 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-23T08:46:36.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:36 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/3177171111' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-23T08:46:36.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:36 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/120744383' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c28ec1b9-2f5f-48e6-8322-7092bbe8cdda"}]: dispatch 2024-01-23T08:46:36.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:36 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/120744383' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c28ec1b9-2f5f-48e6-8322-7092bbe8cdda"}]': finished 2024-01-23T08:46:36.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:36 smithi137 ceph-mon[35787]: osdmap e11: 2 total, 1 up, 2 in 2024-01-23T08:46:36.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:36 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-23T08:46:36.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:36 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/3177171111' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-23T08:46:37.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:37 smithi059 ceph-mon[28424]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:37.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:37 smithi137 ceph-mon[35787]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:38.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:38 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:38.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:38 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:38.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:38 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:46:38.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:38 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:46:38.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:38 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:38.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:38 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:38.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:38 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:38.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:38 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:46:38.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:38 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:46:38.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:38 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:39.789 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:39 smithi059 ceph-mon[28424]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:39.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:39 smithi137 ceph-mon[35787]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:41.291 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:41 smithi059 ceph-mon[28424]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:41.291 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:41 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:46:41.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:41 smithi137 ceph-mon[35787]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:41.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:41 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:46:43.566 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:43 smithi059 ceph-mon[28424]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:43.566 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:43 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-01-23T08:46:43.566 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:43 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:46:43.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:43 smithi137 ceph-mon[35787]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:43.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:43 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-01-23T08:46:43.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:43 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:46:44.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:44 smithi059 ceph-mon[28424]: Deploying daemon osd.1 on smithi059 2024-01-23T08:46:44.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:44 smithi137 ceph-mon[35787]: Deploying daemon osd.1 on smithi059 2024-01-23T08:46:45.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:45 smithi059 ceph-mon[28424]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:45.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:45 smithi137 ceph-mon[35787]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:47.412 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:47 smithi059 ceph-mon[28424]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:47.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:47 smithi137 ceph-mon[35787]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:49.049 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:48 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:49.050 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:48 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:46:49.050 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:48 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:49.050 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:48 smithi059 ceph-mon[28424]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:49.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:48 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:49.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:48 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:46:49.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:48 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:49.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:48 smithi137 ceph-mon[35787]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:51.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:50 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:51.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:50 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:51.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:50 smithi137 ceph-mon[35787]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:51.132 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:50 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:51.132 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:50 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:51.132 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:50 smithi059 ceph-mon[28424]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:51.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:51.893+0000 7f1b967fc700 1 -- 172.21.15.59:0/4174337808 <== mgr.14210 v2:172.21.15.59:6800/3219735162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1ba4105780 con 0x7f1b90068ed0 2024-01-23T08:46:51.891 INFO:teuthology.orchestra.run.smithi059.stdout:Created osd(s) 1 on host 'smithi059' 2024-01-23T08:46:51.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:51.895+0000 7f1bac228700 1 -- 172.21.15.59:0/4174337808 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f1b90068ed0 msgr2=0x7f1b9006b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:46:51.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:51.895+0000 7f1bac228700 1 --2- 172.21.15.59:0/4174337808 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f1b90068ed0 0x7f1b9006b380 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1b98007fd0 tx=0x7f1b9800b040 comp rx=0 tx=0).stop 2024-01-23T08:46:51.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:51.895+0000 7f1bac228700 1 -- 172.21.15.59:0/4174337808 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ba40fe490 msgr2=0x7f1ba40fdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:46:51.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:51.895+0000 7f1bac228700 1 --2- 172.21.15.59:0/4174337808 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ba40fe490 0x7f1ba40fdcd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1ba00134a0 tx=0x7f1ba0009e30 comp rx=0 tx=0).stop 2024-01-23T08:46:51.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:51.895+0000 7f1bac228700 1 -- 172.21.15.59:0/4174337808 shutdown_connections 2024-01-23T08:46:51.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:51.895+0000 7f1bac228700 1 --2- 172.21.15.59:0/4174337808 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f1b90068ed0 0x7f1b9006b380 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:46:51.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:51.895+0000 7f1bac228700 1 --2- 172.21.15.59:0/4174337808 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ba40fe490 0x7f1ba40fdcd0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:46:51.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:51.895+0000 7f1bac228700 1 --2- 172.21.15.59:0/4174337808 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ba40fede0 0x7f1ba40fc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:46:51.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:51.895+0000 7f1bac228700 1 -- 172.21.15.59:0/4174337808 >> 172.21.15.59:0/4174337808 conn(0x7f1ba40f6bb0 msgr2=0x7f1ba4101d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:46:51.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:51.896+0000 7f1bac228700 1 -- 172.21.15.59:0/4174337808 shutdown_connections 2024-01-23T08:46:51.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:46:51.896+0000 7f1bac228700 1 -- 172.21.15.59:0/4174337808 wait complete. 2024-01-23T08:46:53.086 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:52 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:53.086 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:52 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:53.086 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:52 smithi059 ceph-mon[28424]: from='osd.1 [v2:172.21.15.59:6810/1673825133,v1:172.21.15.59:6811/1673825133]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-01-23T08:46:53.086 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:52 smithi059 ceph-mon[28424]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:53.113 DEBUG:teuthology.orchestra.run.smithi059:osd.1> sudo journalctl -f -n 0 -u ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@osd.1.service 2024-01-23T08:46:53.115 INFO:tasks.cephadm:Deploying osd.2 on smithi059 with /dev/vg_nvme/lv_2... 2024-01-23T08:46:53.116 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-01-23T08:46:53.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:52 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:53.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:52 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:53.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:52 smithi137 ceph-mon[35787]: from='osd.1 [v2:172.21.15.59:6810/1673825133,v1:172.21.15.59:6811/1673825133]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-01-23T08:46:53.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:52 smithi137 ceph-mon[35787]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:53.519 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:46:54.126 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:53 smithi059 ceph-mon[28424]: from='osd.1 [v2:172.21.15.59:6810/1673825133,v1:172.21.15.59:6811/1673825133]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-01-23T08:46:54.126 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:53 smithi059 ceph-mon[28424]: osdmap e12: 2 total, 1 up, 2 in 2024-01-23T08:46:54.126 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:53 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-23T08:46:54.127 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:53 smithi059 ceph-mon[28424]: from='osd.1 [v2:172.21.15.59:6810/1673825133,v1:172.21.15.59:6811/1673825133]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-01-23T08:46:54.127 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:46:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:46:53.879+0000 7f27e0bbb700 -1 osd.1 0 waiting for initial osdmap 2024-01-23T08:46:54.127 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:46:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:46:53.883+0000 7f27dd1b9700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-23T08:46:54.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:53 smithi137 ceph-mon[35787]: from='osd.1 [v2:172.21.15.59:6810/1673825133,v1:172.21.15.59:6811/1673825133]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-01-23T08:46:54.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:53 smithi137 ceph-mon[35787]: osdmap e12: 2 total, 1 up, 2 in 2024-01-23T08:46:54.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:53 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-23T08:46:54.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:53 smithi137 ceph-mon[35787]: from='osd.1 [v2:172.21.15.59:6810/1673825133,v1:172.21.15.59:6811/1673825133]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-01-23T08:46:54.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:54 smithi059 ceph-mon[28424]: from='osd.1 [v2:172.21.15.59:6810/1673825133,v1:172.21.15.59:6811/1673825133]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2024-01-23T08:46:54.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:54 smithi059 ceph-mon[28424]: osdmap e13: 2 total, 1 up, 2 in 2024-01-23T08:46:54.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:54 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-23T08:46:54.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:54 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-23T08:46:54.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:54 smithi059 ceph-mon[28424]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:55.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:54 smithi137 ceph-mon[35787]: from='osd.1 [v2:172.21.15.59:6810/1673825133,v1:172.21.15.59:6811/1673825133]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2024-01-23T08:46:55.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:54 smithi137 ceph-mon[35787]: osdmap e13: 2 total, 1 up, 2 in 2024-01-23T08:46:55.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:54 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-23T08:46:55.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:54 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-23T08:46:55.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:54 smithi137 ceph-mon[35787]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-23T08:46:56.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:55 smithi059 ceph-mon[28424]: purged_snaps scrub starts 2024-01-23T08:46:56.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:55 smithi059 ceph-mon[28424]: purged_snaps scrub ok 2024-01-23T08:46:56.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:55 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-23T08:46:56.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:55 smithi059 ceph-mon[28424]: osd.1 [v2:172.21.15.59:6810/1673825133,v1:172.21.15.59:6811/1673825133] boot 2024-01-23T08:46:56.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:55 smithi059 ceph-mon[28424]: osdmap e14: 2 total, 2 up, 2 in 2024-01-23T08:46:56.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:55 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-23T08:46:56.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:55 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:46:56.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:55 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:56.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:55 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:56.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:55 smithi137 ceph-mon[35787]: purged_snaps scrub starts 2024-01-23T08:46:56.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:55 smithi137 ceph-mon[35787]: purged_snaps scrub ok 2024-01-23T08:46:56.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:55 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-23T08:46:56.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:55 smithi137 ceph-mon[35787]: osd.1 [v2:172.21.15.59:6810/1673825133,v1:172.21.15.59:6811/1673825133] boot 2024-01-23T08:46:56.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:55 smithi137 ceph-mon[35787]: osdmap e14: 2 total, 2 up, 2 in 2024-01-23T08:46:56.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:55 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-23T08:46:56.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:55 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:46:56.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:55 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:56.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:55 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:46:56.425 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-01-23T08:46:56.425 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-01-23T08:46:56.425 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10+0 records in 2024-01-23T08:46:56.425 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2024-01-23T08:46:56.425 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012323 s, 851 MB/s 2024-01-23T08:46:56.425 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping successful for: 2024-01-23T08:46:57.191 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:56 smithi059 ceph-mon[28424]: osdmap e15: 2 total, 2 up, 2 in 2024-01-23T08:46:57.191 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:56 smithi059 ceph-mon[28424]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:46:57.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:56 smithi137 ceph-mon[35787]: osdmap e15: 2 total, 2 up, 2 in 2024-01-23T08:46:57.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:56 smithi137 ceph-mon[35787]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:46:57.478 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi059:vg_nvme/lv_2 2024-01-23T08:46:58.443 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:46:59.680 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:46:59 smithi059 ceph-mon[28424]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:46:59.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:46:59 smithi137 ceph-mon[35787]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:00.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.585+0000 7fceaaad2700 1 -- 172.21.15.59:0/323729099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcea41022c0 msgr2=0x7fcea41046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:47:00.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.585+0000 7fceaaad2700 1 --2- 172.21.15.59:0/323729099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcea41022c0 0x7fcea41046a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fce90004940 tx=0x7fce90009e30 comp rx=0 tx=0).stop 2024-01-23T08:47:00.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.585+0000 7fceaaad2700 1 -- 172.21.15.59:0/323729099 shutdown_connections 2024-01-23T08:47:00.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.585+0000 7fceaaad2700 1 --2- 172.21.15.59:0/323729099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcea41022c0 0x7fcea41046a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:47:00.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.585+0000 7fceaaad2700 1 --2- 172.21.15.59:0/323729099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcea4066b00 0x7fcea4101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:47:00.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.585+0000 7fceaaad2700 1 -- 172.21.15.59:0/323729099 >> 172.21.15.59:0/323729099 conn(0x7fcea40f7390 msgr2=0x7fcea40f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:47:00.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.585+0000 7fceaaad2700 1 -- 172.21.15.59:0/323729099 shutdown_connections 2024-01-23T08:47:00.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.585+0000 7fceaaad2700 1 -- 172.21.15.59:0/323729099 wait complete. 2024-01-23T08:47:00.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.586+0000 7fceaaad2700 1 Processor -- start 2024-01-23T08:47:00.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.586+0000 7fceaaad2700 1 -- start start 2024-01-23T08:47:00.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.586+0000 7fceaaad2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcea4066b00 0x7fcea410f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:47:00.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.586+0000 7fceaaad2700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcea41022c0 0x7fcea410fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:47:00.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.586+0000 7fceaaad2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea41103d0 con 0x7fcea4066b00 2024-01-23T08:47:00.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.586+0000 7fceaaad2700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea4110c70 con 0x7fcea41022c0 2024-01-23T08:47:00.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.586+0000 7fcea886e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcea4066b00 0x7fcea410f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:47:00.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.586+0000 7fcea886e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcea4066b00 0x7fcea410f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35272/0 (socket says 172.21.15.59:35272) 2024-01-23T08:47:00.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.586+0000 7fcea886e700 1 -- 172.21.15.59:0/1379956167 learned_addr learned my addr 172.21.15.59:0/1379956167 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:47:00.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.586+0000 7fcea3fff700 1 --2- 172.21.15.59:0/1379956167 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcea41022c0 0x7fcea410fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:47:00.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.587+0000 7fcea886e700 1 -- 172.21.15.59:0/1379956167 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcea41022c0 msgr2=0x7fcea410fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:47:00.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.587+0000 7fcea886e700 1 --2- 172.21.15.59:0/1379956167 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcea41022c0 0x7fcea410fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:47:00.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.587+0000 7fcea886e700 1 -- 172.21.15.59:0/1379956167 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce90005040 con 0x7fcea4066b00 2024-01-23T08:47:00.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.587+0000 7fcea886e700 1 --2- 172.21.15.59:0/1379956167 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcea4066b00 0x7fcea410f890 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fce9800eb40 tx=0x7fce9800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:47:00.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.587+0000 7fcea17fa700 1 -- 172.21.15.59:0/1379956167 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce98007cc0 con 0x7fcea4066b00 2024-01-23T08:47:00.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.587+0000 7fcea17fa700 1 -- 172.21.15.59:0/1379956167 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce9800f040 con 0x7fcea4066b00 2024-01-23T08:47:00.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.587+0000 7fceaaad2700 1 -- 172.21.15.59:0/1379956167 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea4110ef0 con 0x7fcea4066b00 2024-01-23T08:47:00.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.587+0000 7fcea17fa700 1 -- 172.21.15.59:0/1379956167 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce9800c990 con 0x7fcea4066b00 2024-01-23T08:47:00.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.587+0000 7fceaaad2700 1 -- 172.21.15.59:0/1379956167 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea4111430 con 0x7fcea4066b00 2024-01-23T08:47:00.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.588+0000 7fceaaad2700 1 -- 172.21.15.59:0/1379956167 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea404bd60 con 0x7fcea4066b00 2024-01-23T08:47:00.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.589+0000 7fcea17fa700 1 -- 172.21.15.59:0/1379956167 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7fce9800caf0 con 0x7fcea4066b00 2024-01-23T08:47:00.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.589+0000 7fcea17fa700 1 --2- 172.21.15.59:0/1379956167 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fce94068ed0 0x7fce9406b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:47:00.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.589+0000 7fcea17fa700 1 -- 172.21.15.59:0/1379956167 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7fce98085bd0 con 0x7fcea4066b00 2024-01-23T08:47:00.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.589+0000 7fcea3fff700 1 --2- 172.21.15.59:0/1379956167 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fce94068ed0 0x7fce9406b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:47:00.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.590+0000 7fcea3fff700 1 --2- 172.21.15.59:0/1379956167 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fce94068ed0 0x7fce9406b380 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fce90013680 tx=0x7fce90009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:47:00.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.594+0000 7fcea17fa700 1 -- 172.21.15.59:0/1379956167 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fce9804da00 con 0x7fcea4066b00 2024-01-23T08:47:00.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:00.816+0000 7fceaaad2700 1 -- 172.21.15.59:0/1379956167 --> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fcea41118a0 con 0x7fce94068ed0 2024-01-23T08:47:01.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:01 smithi059 ceph-mon[28424]: Detected new or changed devices on smithi059 2024-01-23T08:47:01.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:01 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:01.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:01 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:01.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:01 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:47:01.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:01 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:47:01.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:01 smithi059 ceph-mon[28424]: Adjusting osd_memory_target on smithi059 to 6434M 2024-01-23T08:47:01.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:01 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:01.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:01 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:47:01.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:01 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:47:01.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:01 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:01.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:01 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:47:01.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:01 smithi059 ceph-mon[28424]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:01.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:01 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-23T08:47:01.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:01 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-23T08:47:01.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:01 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:47:01.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:01 smithi137 ceph-mon[35787]: Detected new or changed devices on smithi059 2024-01-23T08:47:01.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:01 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:01.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:01 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:01.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:01 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:47:01.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:01 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:47:01.544 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:01 smithi137 ceph-mon[35787]: Adjusting osd_memory_target on smithi059 to 6434M 2024-01-23T08:47:01.544 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:01 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:01.544 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:01 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:47:01.544 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:01 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:47:01.544 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:01 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:01.544 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:01 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:47:01.544 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:01 smithi137 ceph-mon[35787]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:01.544 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:01 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-23T08:47:01.544 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:01 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-23T08:47:01.544 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:01 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:47:02.456 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:02 smithi059 ceph-mon[28424]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:47:02.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:02 smithi137 ceph-mon[35787]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:47:03.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:03 smithi059 ceph-mon[28424]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:03.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:03 smithi137 ceph-mon[35787]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:04.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:04 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/4000799229' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94d9e85d-6f22-4e83-9f4c-b326e9375e30"}]: dispatch 2024-01-23T08:47:04.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:04 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/4000799229' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94d9e85d-6f22-4e83-9f4c-b326e9375e30"}]': finished 2024-01-23T08:47:04.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:04 smithi059 ceph-mon[28424]: osdmap e16: 3 total, 2 up, 3 in 2024-01-23T08:47:04.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:04 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-23T08:47:04.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:04 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/4000799229' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94d9e85d-6f22-4e83-9f4c-b326e9375e30"}]: dispatch 2024-01-23T08:47:04.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:04 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/4000799229' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94d9e85d-6f22-4e83-9f4c-b326e9375e30"}]': finished 2024-01-23T08:47:04.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:04 smithi137 ceph-mon[35787]: osdmap e16: 3 total, 2 up, 3 in 2024-01-23T08:47:04.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:04 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-23T08:47:05.434 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:05 smithi059 ceph-mon[28424]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:05.434 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:05 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/1866230989' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-23T08:47:05.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:05 smithi137 ceph-mon[35787]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:05.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:05 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/1866230989' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-23T08:47:07.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:06 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:07.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:06 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:07.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:06 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:47:07.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:06 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:47:07.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:06 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:07.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:06 smithi059 ceph-mon[28424]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:07.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:06 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:07.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:06 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:07.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:06 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:47:07.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:06 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:47:07.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:06 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:07.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:06 smithi137 ceph-mon[35787]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:09.601 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:09 smithi059 ceph-mon[28424]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:09.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:09 smithi137 ceph-mon[35787]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:11.267 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:11 smithi059 ceph-mon[28424]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:11.267 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:11 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:47:11.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:11 smithi137 ceph-mon[35787]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:11.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:11 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:47:12.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:12 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-01-23T08:47:12.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:12 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:47:12.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:12 smithi137 ceph-mon[35787]: Deploying daemon osd.2 on smithi059 2024-01-23T08:47:12.433 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:12 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-01-23T08:47:12.433 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:12 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:47:12.433 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:12 smithi059 ceph-mon[28424]: Deploying daemon osd.2 on smithi059 2024-01-23T08:47:13.373 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:13 smithi059 ceph-mon[28424]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:13.462 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:13 smithi137 ceph-mon[35787]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:15.617 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:15 smithi059 ceph-mon[28424]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:15.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:15 smithi137 ceph-mon[35787]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:17.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:17 smithi059 ceph-mon[28424]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:17.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:17 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:17.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:17 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:47:17.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:17 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:17.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:17 smithi137 ceph-mon[35787]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:17.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:17 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:17.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:17 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:47:17.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:17 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:19.518 INFO:teuthology.orchestra.run.smithi059.stdout:Created osd(s) 2 on host 'smithi059' 2024-01-23T08:47:19.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:19.520+0000 7fcea17fa700 1 -- 172.21.15.59:0/1379956167 <== mgr.14210 v2:172.21.15.59:6800/3219735162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcea41118a0 con 0x7fce94068ed0 2024-01-23T08:47:19.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:19.522+0000 7fceaaad2700 1 -- 172.21.15.59:0/1379956167 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fce94068ed0 msgr2=0x7fce9406b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:47:19.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:19.522+0000 7fceaaad2700 1 --2- 172.21.15.59:0/1379956167 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fce94068ed0 0x7fce9406b380 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fce90013680 tx=0x7fce90009e30 comp rx=0 tx=0).stop 2024-01-23T08:47:19.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:19.522+0000 7fceaaad2700 1 -- 172.21.15.59:0/1379956167 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcea4066b00 msgr2=0x7fcea410f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:47:19.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:19.522+0000 7fceaaad2700 1 --2- 172.21.15.59:0/1379956167 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcea4066b00 0x7fcea410f890 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fce9800eb40 tx=0x7fce9800b040 comp rx=0 tx=0).stop 2024-01-23T08:47:19.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:19.523+0000 7fceaaad2700 1 -- 172.21.15.59:0/1379956167 shutdown_connections 2024-01-23T08:47:19.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:19.523+0000 7fceaaad2700 1 --2- 172.21.15.59:0/1379956167 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fce94068ed0 0x7fce9406b380 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:47:19.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:19.523+0000 7fceaaad2700 1 --2- 172.21.15.59:0/1379956167 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcea4066b00 0x7fcea410f890 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:47:19.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:19.523+0000 7fceaaad2700 1 --2- 172.21.15.59:0/1379956167 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcea41022c0 0x7fcea410fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:47:19.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:19.523+0000 7fceaaad2700 1 -- 172.21.15.59:0/1379956167 >> 172.21.15.59:0/1379956167 conn(0x7fcea40f7390 msgr2=0x7fcea40f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:47:19.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:19.523+0000 7fceaaad2700 1 -- 172.21.15.59:0/1379956167 shutdown_connections 2024-01-23T08:47:19.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:47:19.523+0000 7fceaaad2700 1 -- 172.21.15.59:0/1379956167 wait complete. 2024-01-23T08:47:19.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:19 smithi059 ceph-mon[28424]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:19.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:19 smithi137 ceph-mon[35787]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:20.220 DEBUG:teuthology.orchestra.run.smithi059:osd.2> sudo journalctl -f -n 0 -u ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@osd.2.service 2024-01-23T08:47:20.223 INFO:tasks.cephadm:Deploying osd.3 on smithi137 with /dev/vg_nvme/lv_4... 2024-01-23T08:47:20.223 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-01-23T08:47:20.456 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:47:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:47:20.357+0000 7fc62cd49540 -1 osd.2 0 log_to_monitors true 2024-01-23T08:47:20.483 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi137/config 2024-01-23T08:47:20.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:20 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:20.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:20 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:20.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:20 smithi137 ceph-mon[35787]: from='osd.2 [v2:172.21.15.59:6818/2143964017,v1:172.21.15.59:6819/2143964017]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-01-23T08:47:20.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:20 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:20.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:20 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:20.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:20 smithi059 ceph-mon[28424]: from='osd.2 [v2:172.21.15.59:6818/2143964017,v1:172.21.15.59:6819/2143964017]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-01-23T08:47:21.729 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:21 smithi137 ceph-mon[35787]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:21.729 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:21 smithi137 ceph-mon[35787]: from='osd.2 [v2:172.21.15.59:6818/2143964017,v1:172.21.15.59:6819/2143964017]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-01-23T08:47:21.729 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:21 smithi137 ceph-mon[35787]: osdmap e17: 3 total, 2 up, 3 in 2024-01-23T08:47:21.729 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:21 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-23T08:47:21.729 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:21 smithi137 ceph-mon[35787]: from='osd.2 [v2:172.21.15.59:6818/2143964017,v1:172.21.15.59:6819/2143964017]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-01-23T08:47:21.768 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:21 smithi059 ceph-mon[28424]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:21.768 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:21 smithi059 ceph-mon[28424]: from='osd.2 [v2:172.21.15.59:6818/2143964017,v1:172.21.15.59:6819/2143964017]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-01-23T08:47:21.768 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:21 smithi059 ceph-mon[28424]: osdmap e17: 3 total, 2 up, 3 in 2024-01-23T08:47:21.768 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:21 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-23T08:47:21.769 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:21 smithi059 ceph-mon[28424]: from='osd.2 [v2:172.21.15.59:6818/2143964017,v1:172.21.15.59:6819/2143964017]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2024-01-23T08:47:22.333 INFO:teuthology.orchestra.run.smithi137.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-01-23T08:47:22.333 INFO:teuthology.orchestra.run.smithi137.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-01-23T08:47:22.333 INFO:teuthology.orchestra.run.smithi137.stderr: stderr: 10+0 records in 2024-01-23T08:47:22.333 INFO:teuthology.orchestra.run.smithi137.stderr:10+0 records out 2024-01-23T08:47:22.333 INFO:teuthology.orchestra.run.smithi137.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112604 s, 931 MB/s 2024-01-23T08:47:22.333 INFO:teuthology.orchestra.run.smithi137.stderr:--> Zapping successful for: 2024-01-23T08:47:22.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:47:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:47:22.021+0000 7fc62351c700 -1 osd.2 0 waiting for initial osdmap 2024-01-23T08:47:22.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:47:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:47:22.026+0000 7fc61c313700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-23T08:47:22.927 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi137:vg_nvme/lv_4 2024-01-23T08:47:23.179 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi137/config 2024-01-23T08:47:23.286 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:23 smithi137 ceph-mon[35787]: from='osd.2 [v2:172.21.15.59:6818/2143964017,v1:172.21.15.59:6819/2143964017]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2024-01-23T08:47:23.286 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:23 smithi137 ceph-mon[35787]: osdmap e18: 3 total, 2 up, 3 in 2024-01-23T08:47:23.286 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:23 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-23T08:47:23.286 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:23 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-23T08:47:23.286 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:23 smithi137 ceph-mon[35787]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:23.464 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:23 smithi059 ceph-mon[28424]: from='osd.2 [v2:172.21.15.59:6818/2143964017,v1:172.21.15.59:6819/2143964017]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2024-01-23T08:47:23.465 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:23 smithi059 ceph-mon[28424]: osdmap e18: 3 total, 2 up, 3 in 2024-01-23T08:47:23.465 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:23 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-23T08:47:23.465 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:23 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-23T08:47:23.465 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:23 smithi059 ceph-mon[28424]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-23T08:47:24.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:24 smithi137 ceph-mon[35787]: purged_snaps scrub starts 2024-01-23T08:47:24.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:24 smithi137 ceph-mon[35787]: purged_snaps scrub ok 2024-01-23T08:47:24.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:24 smithi137 ceph-mon[35787]: osd.2 [v2:172.21.15.59:6818/2143964017,v1:172.21.15.59:6819/2143964017] boot 2024-01-23T08:47:24.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:24 smithi137 ceph-mon[35787]: osdmap e19: 3 total, 3 up, 3 in 2024-01-23T08:47:24.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:24 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-23T08:47:24.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:24 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:24.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:24 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:24.371 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:24 smithi059 ceph-mon[28424]: purged_snaps scrub starts 2024-01-23T08:47:24.371 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:24 smithi059 ceph-mon[28424]: purged_snaps scrub ok 2024-01-23T08:47:24.371 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:24 smithi059 ceph-mon[28424]: osd.2 [v2:172.21.15.59:6818/2143964017,v1:172.21.15.59:6819/2143964017] boot 2024-01-23T08:47:24.371 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:24 smithi059 ceph-mon[28424]: osdmap e19: 3 total, 3 up, 3 in 2024-01-23T08:47:24.371 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:24 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-23T08:47:24.371 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:24 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:24.371 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:24 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:24.562 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.562+0000 7f6793f96700 1 -- 172.21.15.137:0/2623901914 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f678c0fdfe0 msgr2=0x7f678c0fe430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:47:24.562 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.562+0000 7f6793f96700 1 --2- 172.21.15.137:0/2623901914 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f678c0fdfe0 0x7f678c0fe430 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6788004940 tx=0x7f6788009e30 comp rx=0 tx=0).stop 2024-01-23T08:47:24.562 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.562+0000 7f6793f96700 1 -- 172.21.15.137:0/2623901914 shutdown_connections 2024-01-23T08:47:24.562 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.562+0000 7f6793f96700 1 --2- 172.21.15.137:0/2623901914 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f678c0fdfe0 0x7f678c0fe430 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:47:24.562 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.562+0000 7f6793f96700 1 --2- 172.21.15.137:0/2623901914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f678c0fcdd0 0x7f678c0fd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:47:24.562 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.562+0000 7f6793f96700 1 -- 172.21.15.137:0/2623901914 >> 172.21.15.137:0/2623901914 conn(0x7f678c0f84b0 msgr2=0x7f678c0fa8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:47:24.562 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.562+0000 7f6793f96700 1 -- 172.21.15.137:0/2623901914 shutdown_connections 2024-01-23T08:47:24.562 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.562+0000 7f6793f96700 1 -- 172.21.15.137:0/2623901914 wait complete. 2024-01-23T08:47:24.562 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.563+0000 7f6793f96700 1 Processor -- start 2024-01-23T08:47:24.563 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.563+0000 7f6793f96700 1 -- start start 2024-01-23T08:47:24.563 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.563+0000 7f6793f96700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f678c0fcdd0 0x7f678c071cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:47:24.563 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.563+0000 7f6793f96700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f678c0fdfe0 0x7f678c070170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:47:24.563 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.563+0000 7f6793f96700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c070770 con 0x7f678c0fcdd0 2024-01-23T08:47:24.563 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.563+0000 7f6793f96700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c0708e0 con 0x7f678c0fdfe0 2024-01-23T08:47:24.563 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.563+0000 7f6791531700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f678c0fdfe0 0x7f678c070170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:47:24.563 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.563+0000 7f6791531700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f678c0fdfe0 0x7f678c070170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.137:41688/0 (socket says 172.21.15.137:41688) 2024-01-23T08:47:24.563 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.563+0000 7f6791531700 1 -- 172.21.15.137:0/3345049075 learned_addr learned my addr 172.21.15.137:0/3345049075 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-23T08:47:24.563 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.563+0000 7f6791d32700 1 --2- 172.21.15.137:0/3345049075 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f678c0fcdd0 0x7f678c071cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:47:24.563 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.563+0000 7f6791531700 1 -- 172.21.15.137:0/3345049075 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f678c0fcdd0 msgr2=0x7f678c071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:47:24.563 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.563+0000 7f6791531700 1 --2- 172.21.15.137:0/3345049075 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f678c0fcdd0 0x7f678c071cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:47:24.563 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.563+0000 7f6791531700 1 -- 172.21.15.137:0/3345049075 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6788005040 con 0x7f678c0fdfe0 2024-01-23T08:47:24.563 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.564+0000 7f6791531700 1 --2- 172.21.15.137:0/3345049075 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f678c0fdfe0 0x7f678c070170 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6788009360 tx=0x7f6788009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:47:24.563 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.564+0000 7f67827fc700 1 -- 172.21.15.137:0/3345049075 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f678800e040 con 0x7f678c0fdfe0 2024-01-23T08:47:24.563 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.564+0000 7f6793f96700 1 -- 172.21.15.137:0/3345049075 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f678c070b60 con 0x7f678c0fdfe0 2024-01-23T08:47:24.564 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.564+0000 7f67827fc700 1 -- 172.21.15.137:0/3345049075 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f678800bc50 con 0x7f678c0fdfe0 2024-01-23T08:47:24.564 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.564+0000 7f67827fc700 1 -- 172.21.15.137:0/3345049075 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f678801c3f0 con 0x7f678c0fdfe0 2024-01-23T08:47:24.564 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.564+0000 7f6793f96700 1 -- 172.21.15.137:0/3345049075 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f678c115810 con 0x7f678c0fdfe0 2024-01-23T08:47:24.565 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.565+0000 7f6793f96700 1 -- 172.21.15.137:0/3345049075 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f678c107700 con 0x7f678c0fdfe0 2024-01-23T08:47:24.565 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.565+0000 7f67827fc700 1 -- 172.21.15.137:0/3345049075 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7f6788013420 con 0x7f678c0fdfe0 2024-01-23T08:47:24.565 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.566+0000 7f67827fc700 1 --2- 172.21.15.137:0/3345049075 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f6778068ed0 0x7f677806b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:47:24.565 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.566+0000 7f67827fc700 1 -- 172.21.15.137:0/3345049075 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 2842+0+0 (secure 0 0 0) 0x7f67880874c0 con 0x7f678c0fdfe0 2024-01-23T08:47:24.565 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.566+0000 7f6791d32700 1 --2- 172.21.15.137:0/3345049075 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f6778068ed0 0x7f677806b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:47:24.565 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.566+0000 7f6791d32700 1 --2- 172.21.15.137:0/3345049075 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f6778068ed0 0x7f677806b380 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f678c071a80 tx=0x7f677c00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:47:24.568 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.570+0000 7f67827fc700 1 -- 172.21.15.137:0/3345049075 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6788057f40 con 0x7f678c0fdfe0 2024-01-23T08:47:24.756 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:24.759+0000 7f6793f96700 1 -- 172.21.15.137:0/3345049075 --> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f678c02b270 con 0x7f6778068ed0 2024-01-23T08:47:25.363 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:25 smithi137 ceph-mon[35787]: osdmap e20: 3 total, 3 up, 3 in 2024-01-23T08:47:25.363 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:25 smithi137 ceph-mon[35787]: pgmap v75: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-01-23T08:47:25.363 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:25 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-01-23T08:47:25.363 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:25 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-23T08:47:25.363 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:25 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-23T08:47:25.363 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:25 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:47:25.418 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:25 smithi059 ceph-mon[28424]: osdmap e20: 3 total, 3 up, 3 in 2024-01-23T08:47:25.418 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:25 smithi059 ceph-mon[28424]: pgmap v75: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-01-23T08:47:25.418 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:25 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-01-23T08:47:25.418 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:25 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-23T08:47:25.418 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:25 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-23T08:47:25.419 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:25 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:47:26.332 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:26 smithi137 ceph-mon[35787]: from='client.24125 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:47:26.332 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:26 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-01-23T08:47:26.332 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:26 smithi137 ceph-mon[35787]: osdmap e21: 3 total, 3 up, 3 in 2024-01-23T08:47:26.332 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:26 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-01-23T08:47:26.332 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:26 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:47:26.332 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:26 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:26.332 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:26 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:26.333 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:26 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:47:26.333 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:26 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:47:26.333 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:26 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:47:26.333 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:26 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:26.333 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:26 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:47:26.333 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:26 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:47:26.333 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:26 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:26.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:26 smithi059 ceph-mon[28424]: from='client.24125 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:47:26.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:26 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-01-23T08:47:26.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:26 smithi059 ceph-mon[28424]: osdmap e21: 3 total, 3 up, 3 in 2024-01-23T08:47:26.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:26 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-01-23T08:47:26.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:26 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:47:26.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:26 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:26.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:26 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:26.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:26 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:47:26.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:26 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:47:26.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:26 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:47:26.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:26 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:26.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:26 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:47:26.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:26 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:47:26.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:26 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:27.078 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:47:26 smithi059 sudo[65979]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-23T08:47:27.078 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:47:26 smithi059 sudo[65979]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-23T08:47:27.078 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:47:26 smithi059 sudo[65979]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-23T08:47:27.078 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:47:26 smithi059 sudo[65979]: pam_unix(sudo:session): session closed for user root 2024-01-23T08:47:27.378 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:47:27 smithi059 sudo[65996]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-23T08:47:27.378 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:47:27 smithi059 sudo[65996]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-23T08:47:27.378 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:47:27 smithi059 sudo[65996]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-23T08:47:27.378 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:47:27 smithi059 sudo[65996]: pam_unix(sudo:session): session closed for user root 2024-01-23T08:47:27.378 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:27 smithi059 ceph-mon[28424]: Detected new or changed devices on smithi059 2024-01-23T08:47:27.378 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:27 smithi059 ceph-mon[28424]: Adjusting osd_memory_target on smithi059 to 4289M 2024-01-23T08:47:27.378 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:27 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-01-23T08:47:27.379 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:27 smithi059 ceph-mon[28424]: osdmap e22: 3 total, 3 up, 3 in 2024-01-23T08:47:27.379 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:27 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:47:27.379 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:27 smithi059 ceph-mon[28424]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-01-23T08:47:27.455 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:27 smithi137 ceph-mon[35787]: Detected new or changed devices on smithi059 2024-01-23T08:47:27.455 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:27 smithi137 ceph-mon[35787]: Adjusting osd_memory_target on smithi059 to 4289M 2024-01-23T08:47:27.455 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:27 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-01-23T08:47:27.455 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:27 smithi137 ceph-mon[35787]: osdmap e22: 3 total, 3 up, 3 in 2024-01-23T08:47:27.455 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:27 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:47:27.455 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:27 smithi137 ceph-mon[35787]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-01-23T08:47:27.648 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:47:27 smithi059 sudo[66067]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-23T08:47:27.648 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:47:27 smithi059 sudo[66067]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-23T08:47:27.649 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:47:27 smithi059 sudo[66067]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-23T08:47:27.649 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:47:27 smithi059 sudo[66067]: pam_unix(sudo:session): session closed for user root 2024-01-23T08:47:27.905 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:47:27 smithi059 sudo[66105]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-23T08:47:27.906 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:47:27 smithi059 sudo[66105]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-23T08:47:27.906 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:47:27 smithi059 sudo[66105]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-23T08:47:27.906 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:47:27 smithi059 sudo[66105]: pam_unix(sudo:session): session closed for user root 2024-01-23T08:47:28.125 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:28.127+0000 7f67827fc700 1 -- 172.21.15.137:0/3345049075 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f6788054880 con 0x7f678c0fdfe0 2024-01-23T08:47:28.175 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:47:27 smithi059 sudo[66110]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-23T08:47:28.175 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:47:27 smithi059 sudo[66110]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-23T08:47:28.175 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:47:27 smithi059 sudo[66110]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-23T08:47:28.175 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:47:27 smithi059 sudo[66110]: pam_unix(sudo:session): session closed for user root 2024-01-23T08:47:28.175 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:28 smithi059 ceph-mon[28424]: osdmap e23: 3 total, 3 up, 3 in 2024-01-23T08:47:28.175 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:28 smithi059 ceph-mon[28424]: from='client.? 172.21.15.137:0/179511962' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "19dfe941-4ce2-4490-8a09-bf85493593bb"}]: dispatch 2024-01-23T08:47:28.176 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:28 smithi059 ceph-mon[28424]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "19dfe941-4ce2-4490-8a09-bf85493593bb"}]: dispatch 2024-01-23T08:47:28.176 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:28 smithi059 ceph-mon[28424]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "19dfe941-4ce2-4490-8a09-bf85493593bb"}]': finished 2024-01-23T08:47:28.176 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:28 smithi059 ceph-mon[28424]: osdmap e24: 4 total, 3 up, 4 in 2024-01-23T08:47:28.176 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:28 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-23T08:47:28.176 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:28 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:28.176 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:28 smithi059 ceph-mon[28424]: from='client.? 172.21.15.137:0/597207227' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-23T08:47:28.434 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:47:28 smithi059 sudo[66159]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-23T08:47:28.434 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:47:28 smithi059 sudo[66159]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-23T08:47:28.434 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:47:28 smithi059 sudo[66159]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-23T08:47:28.434 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:47:28 smithi059 sudo[66159]: pam_unix(sudo:session): session closed for user root 2024-01-23T08:47:28.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:28 smithi137 ceph-mon[35787]: osdmap e23: 3 total, 3 up, 3 in 2024-01-23T08:47:28.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:28 smithi137 ceph-mon[35787]: from='client.? 172.21.15.137:0/179511962' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "19dfe941-4ce2-4490-8a09-bf85493593bb"}]: dispatch 2024-01-23T08:47:28.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:28 smithi137 ceph-mon[35787]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "19dfe941-4ce2-4490-8a09-bf85493593bb"}]: dispatch 2024-01-23T08:47:28.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:28 smithi137 ceph-mon[35787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "19dfe941-4ce2-4490-8a09-bf85493593bb"}]': finished 2024-01-23T08:47:28.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:28 smithi137 ceph-mon[35787]: osdmap e24: 4 total, 3 up, 4 in 2024-01-23T08:47:28.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:28 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-23T08:47:28.544 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:28 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:28.544 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:28 smithi137 ceph-mon[35787]: from='client.? 172.21.15.137:0/597207227' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-23T08:47:28.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:28 smithi059 sudo[66184]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-23T08:47:28.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:28 smithi059 sudo[66184]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-23T08:47:28.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:28 smithi059 sudo[66184]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-23T08:47:28.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:28 smithi059 sudo[66184]: pam_unix(sudo:session): session closed for user root 2024-01-23T08:47:28.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:28 smithi059 sudo[66189]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-23T08:47:28.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:28 smithi059 sudo[66189]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-23T08:47:28.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:28 smithi059 sudo[66189]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-23T08:47:28.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:28 smithi059 sudo[66189]: pam_unix(sudo:session): session closed for user root 2024-01-23T08:47:29.206 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:28 smithi137 sudo[39538]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-23T08:47:29.206 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:28 smithi137 sudo[39538]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-23T08:47:29.206 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:28 smithi137 sudo[39538]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-23T08:47:29.206 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:28 smithi137 sudo[39538]: pam_unix(sudo:session): session closed for user root 2024-01-23T08:47:29.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:29 smithi059 ceph-mon[28424]: mgrmap e19: smithi059.bzqirf(active, since 2m), standbys: smithi137.ooirrh 2024-01-23T08:47:29.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:29 smithi059 ceph-mon[28424]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-23T08:47:29.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:29 smithi059 ceph-mon[28424]: pgmap v81: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-01-23T08:47:29.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:29 smithi059 ceph-mon[28424]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-23T08:47:29.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:29 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-01-23T08:47:29.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:29 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-01-23T08:47:29.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:29 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-01-23T08:47:29.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:29 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-01-23T08:47:29.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:29 smithi137 sudo[39542]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-23T08:47:29.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:29 smithi137 sudo[39542]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-23T08:47:29.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:29 smithi137 sudo[39542]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-23T08:47:29.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:29 smithi137 sudo[39542]: pam_unix(sudo:session): session closed for user root 2024-01-23T08:47:29.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:29 smithi137 ceph-mon[35787]: mgrmap e19: smithi059.bzqirf(active, since 2m), standbys: smithi137.ooirrh 2024-01-23T08:47:29.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:29 smithi137 ceph-mon[35787]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-23T08:47:29.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:29 smithi137 ceph-mon[35787]: pgmap v81: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-01-23T08:47:29.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:29 smithi137 ceph-mon[35787]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-23T08:47:29.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:29 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-01-23T08:47:29.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:29 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-01-23T08:47:29.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:29 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2024-01-23T08:47:29.544 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:29 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-01-23T08:47:30.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:30 smithi059 ceph-mon[28424]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-23T08:47:30.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:30 smithi059 ceph-mon[28424]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-23T08:47:30.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:30 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:30.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:30 smithi137 ceph-mon[35787]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-23T08:47:30.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:30 smithi137 ceph-mon[35787]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-23T08:47:30.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:30 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:31.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:31 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:31.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:31 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:31.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:31 smithi137 ceph-mon[35787]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-01-23T08:47:31.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:31 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:47:31.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:31 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:47:31.544 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:31 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:31.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:31 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:31.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:31 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:31.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:31 smithi059 ceph-mon[28424]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-01-23T08:47:31.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:31 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:47:31.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:31 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:47:31.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:31 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:33.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:33 smithi059 ceph-mon[28424]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-01-23T08:47:33.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:33 smithi137 ceph-mon[35787]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-01-23T08:47:34.635 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:34 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-01-23T08:47:34.635 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:34 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:47:34.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:34 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-01-23T08:47:34.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:34 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:47:35.721 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:35 smithi137 ceph-mon[35787]: Deploying daemon osd.3 on smithi137 2024-01-23T08:47:35.721 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:35 smithi137 ceph-mon[35787]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-01-23T08:47:35.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:35 smithi059 ceph-mon[28424]: Deploying daemon osd.3 on smithi137 2024-01-23T08:47:35.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:35 smithi059 ceph-mon[28424]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-01-23T08:47:37.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:37 smithi059 ceph-mon[28424]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-01-23T08:47:37.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:37 smithi137 ceph-mon[35787]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-01-23T08:47:39.656 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:39 smithi137 ceph-mon[35787]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-01-23T08:47:39.656 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:39 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:39.656 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:39 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:39.656 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:39 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:47:39.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:39 smithi059 ceph-mon[28424]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-01-23T08:47:39.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:39 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:39.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:39 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:39.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:39 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:47:41.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:41 smithi137 ceph-mon[35787]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-01-23T08:47:41.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:41 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:47:41.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:41 smithi059 ceph-mon[28424]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-01-23T08:47:41.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:41 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:47:42.426 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:42.428+0000 7f67827fc700 1 -- 172.21.15.137:0/3345049075 <== mgr.14210 v2:172.21.15.59:6800/3219735162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f678c02b270 con 0x7f6778068ed0 2024-01-23T08:47:42.426 INFO:teuthology.orchestra.run.smithi137.stdout:Created osd(s) 3 on host 'smithi137' 2024-01-23T08:47:42.428 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:42.430+0000 7f6793f96700 1 -- 172.21.15.137:0/3345049075 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f6778068ed0 msgr2=0x7f677806b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:47:42.428 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:42.430+0000 7f6793f96700 1 --2- 172.21.15.137:0/3345049075 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f6778068ed0 0x7f677806b380 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f678c071a80 tx=0x7f677c00b040 comp rx=0 tx=0).stop 2024-01-23T08:47:42.428 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:42.430+0000 7f6793f96700 1 -- 172.21.15.137:0/3345049075 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f678c0fdfe0 msgr2=0x7f678c070170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:47:42.428 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:42.430+0000 7f6793f96700 1 --2- 172.21.15.137:0/3345049075 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f678c0fdfe0 0x7f678c070170 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6788009360 tx=0x7f6788009e30 comp rx=0 tx=0).stop 2024-01-23T08:47:42.428 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:42.430+0000 7f6793f96700 1 -- 172.21.15.137:0/3345049075 shutdown_connections 2024-01-23T08:47:42.428 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:42.430+0000 7f6793f96700 1 --2- 172.21.15.137:0/3345049075 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f6778068ed0 0x7f677806b380 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:47:42.428 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:42.430+0000 7f6793f96700 1 --2- 172.21.15.137:0/3345049075 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f678c0fcdd0 0x7f678c071cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:47:42.428 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:42.430+0000 7f6793f96700 1 --2- 172.21.15.137:0/3345049075 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f678c0fdfe0 0x7f678c070170 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:47:42.429 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:42.430+0000 7f6793f96700 1 -- 172.21.15.137:0/3345049075 >> 172.21.15.137:0/3345049075 conn(0x7f678c0f84b0 msgr2=0x7f678c0fa840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:47:42.429 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:42.430+0000 7f6793f96700 1 -- 172.21.15.137:0/3345049075 shutdown_connections 2024-01-23T08:47:42.429 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:42.430+0000 7f6793f96700 1 -- 172.21.15.137:0/3345049075 wait complete. 2024-01-23T08:47:43.392 DEBUG:teuthology.orchestra.run.smithi137:osd.3> sudo journalctl -f -n 0 -u ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@osd.3.service 2024-01-23T08:47:43.394 INFO:tasks.cephadm:Deploying osd.4 on smithi137 with /dev/vg_nvme/lv_3... 2024-01-23T08:47:43.395 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-01-23T08:47:43.697 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi137/config 2024-01-23T08:47:43.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:43 smithi059 ceph-mon[28424]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-01-23T08:47:43.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:43 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:43.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:43 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:43.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:43 smithi059 ceph-mon[28424]: from='osd.3 [v2:172.21.15.137:6800/2062812031,v1:172.21.15.137:6801/2062812031]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-23T08:47:43.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:43 smithi059 ceph-mon[28424]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-23T08:47:43.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:43 smithi137 ceph-mon[35787]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-01-23T08:47:43.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:43 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:43.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:43 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:43.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:43 smithi137 ceph-mon[35787]: from='osd.3 [v2:172.21.15.137:6800/2062812031,v1:172.21.15.137:6801/2062812031]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-23T08:47:43.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:43 smithi137 ceph-mon[35787]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-23T08:47:44.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:44 smithi059 ceph-mon[28424]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-01-23T08:47:44.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:44 smithi059 ceph-mon[28424]: osdmap e25: 4 total, 3 up, 4 in 2024-01-23T08:47:44.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:44 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-23T08:47:44.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:44 smithi059 ceph-mon[28424]: from='osd.3 [v2:172.21.15.137:6800/2062812031,v1:172.21.15.137:6801/2062812031]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-01-23T08:47:44.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:44 smithi059 ceph-mon[28424]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-01-23T08:47:44.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:44 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:44.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:44 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:44.782 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:44 smithi137 ceph-mon[35787]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-01-23T08:47:44.783 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:44 smithi137 ceph-mon[35787]: osdmap e25: 4 total, 3 up, 4 in 2024-01-23T08:47:44.783 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:44 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-23T08:47:44.783 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:44 smithi137 ceph-mon[35787]: from='osd.3 [v2:172.21.15.137:6800/2062812031,v1:172.21.15.137:6801/2062812031]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-01-23T08:47:44.783 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:44 smithi137 ceph-mon[35787]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-01-23T08:47:44.783 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:44 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:44.783 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:44 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:44.783 INFO:journalctl@ceph.osd.3.smithi137.stdout:Jan 23 08:47:44 smithi137 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-3[41183]: 2024-01-23T08:47:44.422+0000 7f7a60f7b700 -1 osd.3 0 waiting for initial osdmap 2024-01-23T08:47:44.784 INFO:journalctl@ceph.osd.3.smithi137.stdout:Jan 23 08:47:44 smithi137 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-3[41183]: 2024-01-23T08:47:44.428+0000 7f7a5bd76700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-23T08:47:45.634 INFO:teuthology.orchestra.run.smithi137.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-01-23T08:47:45.634 INFO:teuthology.orchestra.run.smithi137.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-01-23T08:47:45.634 INFO:teuthology.orchestra.run.smithi137.stderr: stderr: 10+0 records in 2024-01-23T08:47:45.634 INFO:teuthology.orchestra.run.smithi137.stderr:10+0 records out 2024-01-23T08:47:45.634 INFO:teuthology.orchestra.run.smithi137.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112342 s, 933 MB/s 2024-01-23T08:47:45.634 INFO:teuthology.orchestra.run.smithi137.stderr:--> Zapping successful for: 2024-01-23T08:47:45.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:45 smithi059 ceph-mon[28424]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-01-23T08:47:45.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:45 smithi059 ceph-mon[28424]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-01-23T08:47:45.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:45 smithi059 ceph-mon[28424]: osdmap e26: 4 total, 3 up, 4 in 2024-01-23T08:47:45.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:45 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-23T08:47:45.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:45 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-23T08:47:45.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:45 smithi059 ceph-mon[28424]: osd.3 [v2:172.21.15.137:6800/2062812031,v1:172.21.15.137:6801/2062812031] boot 2024-01-23T08:47:45.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:45 smithi059 ceph-mon[28424]: osdmap e27: 4 total, 4 up, 4 in 2024-01-23T08:47:45.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:45 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-23T08:47:45.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:45 smithi137 ceph-mon[35787]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-01-23T08:47:45.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:45 smithi137 ceph-mon[35787]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-01-23T08:47:45.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:45 smithi137 ceph-mon[35787]: osdmap e26: 4 total, 3 up, 4 in 2024-01-23T08:47:45.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:45 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-23T08:47:45.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:45 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-23T08:47:45.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:45 smithi137 ceph-mon[35787]: osd.3 [v2:172.21.15.137:6800/2062812031,v1:172.21.15.137:6801/2062812031] boot 2024-01-23T08:47:45.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:45 smithi137 ceph-mon[35787]: osdmap e27: 4 total, 4 up, 4 in 2024-01-23T08:47:45.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:45 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-23T08:47:46.316 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi137:vg_nvme/lv_3 2024-01-23T08:47:46.687 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:46 smithi137 ceph-mon[35787]: purged_snaps scrub starts 2024-01-23T08:47:46.687 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:46 smithi137 ceph-mon[35787]: purged_snaps scrub ok 2024-01-23T08:47:46.687 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:46 smithi137 ceph-mon[35787]: osdmap e28: 4 total, 4 up, 4 in 2024-01-23T08:47:46.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:46 smithi059 ceph-mon[28424]: purged_snaps scrub starts 2024-01-23T08:47:46.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:46 smithi059 ceph-mon[28424]: purged_snaps scrub ok 2024-01-23T08:47:46.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:46 smithi059 ceph-mon[28424]: osdmap e28: 4 total, 4 up, 4 in 2024-01-23T08:47:47.001 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi137/config 2024-01-23T08:47:47.677 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:47 smithi137 ceph-mon[35787]: pgmap v94: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-23T08:47:47.677 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:47 smithi137 ceph-mon[35787]: osdmap e29: 4 total, 4 up, 4 in 2024-01-23T08:47:47.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:47 smithi059 ceph-mon[28424]: pgmap v94: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-23T08:47:47.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:47 smithi059 ceph-mon[28424]: osdmap e29: 4 total, 4 up, 4 in 2024-01-23T08:47:49.201 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:49 smithi137 ceph-mon[35787]: osdmap e30: 4 total, 4 up, 4 in 2024-01-23T08:47:49.201 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:49 smithi137 ceph-mon[35787]: pgmap v97: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-23T08:47:49.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:49 smithi059 ceph-mon[28424]: osdmap e30: 4 total, 4 up, 4 in 2024-01-23T08:47:49.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:49 smithi059 ceph-mon[28424]: pgmap v97: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-23T08:47:49.487 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.489+0000 7fd3c139c700 1 -- 172.21.15.137:0/357673634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3bc0fcdd0 msgr2=0x7fd3bc0fd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:47:49.487 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.489+0000 7fd3c139c700 1 --2- 172.21.15.137:0/357673634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3bc0fcdd0 0x7fd3bc0fd1e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac004940 tx=0x7fd3ac009e30 comp rx=0 tx=0).stop 2024-01-23T08:47:49.487 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.489+0000 7fd3c139c700 1 -- 172.21.15.137:0/357673634 shutdown_connections 2024-01-23T08:47:49.487 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.489+0000 7fd3c139c700 1 --2- 172.21.15.137:0/357673634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3bc0fdfe0 0x7fd3bc0fe430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:47:49.487 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.489+0000 7fd3c139c700 1 --2- 172.21.15.137:0/357673634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3bc0fcdd0 0x7fd3bc0fd1e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:47:49.487 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.489+0000 7fd3c139c700 1 -- 172.21.15.137:0/357673634 >> 172.21.15.137:0/357673634 conn(0x7fd3bc0f84b0 msgr2=0x7fd3bc0fa8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:47:49.487 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.489+0000 7fd3c139c700 1 -- 172.21.15.137:0/357673634 shutdown_connections 2024-01-23T08:47:49.487 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.489+0000 7fd3c139c700 1 -- 172.21.15.137:0/357673634 wait complete. 2024-01-23T08:47:49.487 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.489+0000 7fd3c139c700 1 Processor -- start 2024-01-23T08:47:49.488 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.489+0000 7fd3c139c700 1 -- start start 2024-01-23T08:47:49.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.490+0000 7fd3c139c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3bc0fcdd0 0x7fd3bc071cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:47:49.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.490+0000 7fd3c139c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3bc0fdfe0 0x7fd3bc070170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:47:49.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.490+0000 7fd3c139c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc070770 con 0x7fd3bc0fcdd0 2024-01-23T08:47:49.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.490+0000 7fd3c139c700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc0708e0 con 0x7fd3bc0fdfe0 2024-01-23T08:47:49.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.490+0000 7fd3b27fc700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3bc0fdfe0 0x7fd3bc070170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:47:49.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.490+0000 7fd3b27fc700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3bc0fdfe0 0x7fd3bc070170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.137:40708/0 (socket says 172.21.15.137:40708) 2024-01-23T08:47:49.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.490+0000 7fd3b27fc700 1 -- 172.21.15.137:0/4183357734 learned_addr learned my addr 172.21.15.137:0/4183357734 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-23T08:47:49.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.490+0000 7fd3baffd700 1 --2- 172.21.15.137:0/4183357734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3bc0fcdd0 0x7fd3bc071cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:47:49.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.490+0000 7fd3b27fc700 1 -- 172.21.15.137:0/4183357734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3bc0fcdd0 msgr2=0x7fd3bc071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:47:49.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.490+0000 7fd3b27fc700 1 --2- 172.21.15.137:0/4183357734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3bc0fcdd0 0x7fd3bc071cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:47:49.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.490+0000 7fd3b27fc700 1 -- 172.21.15.137:0/4183357734 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ac005040 con 0x7fd3bc0fdfe0 2024-01-23T08:47:49.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.490+0000 7fd3b27fc700 1 --2- 172.21.15.137:0/4183357734 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3bc0fdfe0 0x7fd3bc070170 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400c8c0 tx=0x7fd3a40044c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:47:49.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.490+0000 7fd3b3fff700 1 -- 172.21.15.137:0/4183357734 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3a400a980 con 0x7fd3bc0fdfe0 2024-01-23T08:47:49.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.490+0000 7fd3b3fff700 1 -- 172.21.15.137:0/4183357734 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd3a400ce60 con 0x7fd3bc0fdfe0 2024-01-23T08:47:49.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.490+0000 7fd3c139c700 1 -- 172.21.15.137:0/4183357734 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3bc070bc0 con 0x7fd3bc0fdfe0 2024-01-23T08:47:49.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.490+0000 7fd3b3fff700 1 -- 172.21.15.137:0/4183357734 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3a40113a0 con 0x7fd3bc0fdfe0 2024-01-23T08:47:49.490 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.490+0000 7fd3c139c700 1 -- 172.21.15.137:0/4183357734 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3bc1157e0 con 0x7fd3bc0fdfe0 2024-01-23T08:47:49.490 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.491+0000 7fd3c139c700 1 -- 172.21.15.137:0/4183357734 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3bc1076e0 con 0x7fd3bc0fdfe0 2024-01-23T08:47:49.490 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.492+0000 7fd3b3fff700 1 -- 172.21.15.137:0/4183357734 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fd3a4012400 con 0x7fd3bc0fdfe0 2024-01-23T08:47:49.490 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.492+0000 7fd3b3fff700 1 --2- 172.21.15.137:0/4183357734 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fd3a8068f70 0x7fd3a806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:47:49.490 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.492+0000 7fd3b3fff700 1 -- 172.21.15.137:0/4183357734 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 3770+0+0 (secure 0 0 0) 0x7fd3a4086840 con 0x7fd3bc0fdfe0 2024-01-23T08:47:49.490 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.492+0000 7fd3baffd700 1 --2- 172.21.15.137:0/4183357734 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fd3a8068f70 0x7fd3a806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:47:49.491 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.493+0000 7fd3baffd700 1 --2- 172.21.15.137:0/4183357734 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fd3a8068f70 0x7fd3a806b420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac004f90 tx=0x7fd3ac009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:47:49.494 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.497+0000 7fd3b3fff700 1 -- 172.21.15.137:0/4183357734 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fd3a4056e40 con 0x7fd3bc0fdfe0 2024-01-23T08:47:49.682 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:47:49.685+0000 7fd3c139c700 1 -- 172.21.15.137:0/4183357734 --> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3bc05e260 con 0x7fd3a8068f70 2024-01-23T08:47:50.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:50 smithi137 ceph-mon[35787]: osdmap e31: 4 total, 4 up, 4 in 2024-01-23T08:47:50.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:50 smithi137 ceph-mon[35787]: Detected new or changed devices on smithi137 2024-01-23T08:47:50.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:50 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:50.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:50 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:50.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:50 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:47:50.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:50 smithi137 ceph-mon[35787]: Adjusting osd_memory_target on smithi137 to 15936M 2024-01-23T08:47:50.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:50 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:50.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:50 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:47:50.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:50 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:47:50.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:50 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:50.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:50 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:47:50.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:50 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:47:50.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:50 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:47:50.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:50 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:50.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:50 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-23T08:47:50.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:50 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-23T08:47:50.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:50 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:47:50.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:50 smithi059 ceph-mon[28424]: osdmap e31: 4 total, 4 up, 4 in 2024-01-23T08:47:50.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:50 smithi059 ceph-mon[28424]: Detected new or changed devices on smithi137 2024-01-23T08:47:50.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:50 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:50.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:50 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:50.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:50 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:47:50.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:50 smithi059 ceph-mon[28424]: Adjusting osd_memory_target on smithi137 to 15936M 2024-01-23T08:47:50.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:50 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:50.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:50 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:47:50.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:50 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:47:50.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:50 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:50.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:50 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:47:50.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:50 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:47:50.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:50 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:47:50.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:50 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:47:50.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:50 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-23T08:47:50.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:50 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-23T08:47:50.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:50 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:47:51.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:51 smithi137 ceph-mon[35787]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:47:51.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:51 smithi137 ceph-mon[35787]: pgmap v99: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-23T08:47:51.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:51 smithi059 ceph-mon[28424]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:47:51.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:51 smithi059 ceph-mon[28424]: pgmap v99: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-23T08:47:52.629 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:52 smithi137 ceph-mon[35787]: from='client.? 172.21.15.137:0/4206759001' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a1d6209f-095c-42e2-966a-11559d090531"}]: dispatch 2024-01-23T08:47:52.629 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:52 smithi137 ceph-mon[35787]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a1d6209f-095c-42e2-966a-11559d090531"}]: dispatch 2024-01-23T08:47:52.630 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:52 smithi137 ceph-mon[35787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a1d6209f-095c-42e2-966a-11559d090531"}]': finished 2024-01-23T08:47:52.630 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:52 smithi137 ceph-mon[35787]: osdmap e32: 5 total, 4 up, 5 in 2024-01-23T08:47:52.630 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:52 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-23T08:47:52.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:52 smithi059 ceph-mon[28424]: from='client.? 172.21.15.137:0/4206759001' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a1d6209f-095c-42e2-966a-11559d090531"}]: dispatch 2024-01-23T08:47:52.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:52 smithi059 ceph-mon[28424]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a1d6209f-095c-42e2-966a-11559d090531"}]: dispatch 2024-01-23T08:47:52.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:52 smithi059 ceph-mon[28424]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a1d6209f-095c-42e2-966a-11559d090531"}]': finished 2024-01-23T08:47:52.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:52 smithi059 ceph-mon[28424]: osdmap e32: 5 total, 4 up, 5 in 2024-01-23T08:47:52.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:52 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-23T08:47:53.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:53 smithi059 ceph-mon[28424]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-01-23T08:47:53.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:53 smithi059 ceph-mon[28424]: from='client.? 172.21.15.137:0/3913996207' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-23T08:47:53.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:53 smithi137 ceph-mon[35787]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-01-23T08:47:53.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:53 smithi137 ceph-mon[35787]: from='client.? 172.21.15.137:0/3913996207' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-23T08:47:55.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:55 smithi059 ceph-mon[28424]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-01-23T08:47:55.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:55 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:47:55.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:55 smithi137 ceph-mon[35787]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-01-23T08:47:55.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:55 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:47:57.612 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:57 smithi137 ceph-mon[35787]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-01-23T08:47:57.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:57 smithi059 ceph-mon[28424]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-01-23T08:47:59.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:47:59 smithi137 ceph-mon[35787]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-01-23T08:47:59.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:47:59 smithi059 ceph-mon[28424]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-01-23T08:48:00.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:00 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-01-23T08:48:00.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:00 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:48:00.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:00 smithi137 ceph-mon[35787]: Deploying daemon osd.4 on smithi137 2024-01-23T08:48:00.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:00 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-01-23T08:48:00.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:00 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:48:00.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:00 smithi059 ceph-mon[28424]: Deploying daemon osd.4 on smithi137 2024-01-23T08:48:01.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:01 smithi137 ceph-mon[35787]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-01-23T08:48:01.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:01 smithi059 ceph-mon[28424]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-01-23T08:48:03.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:03 smithi137 ceph-mon[35787]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-23T08:48:03.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:03 smithi059 ceph-mon[28424]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-23T08:48:05.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:05 smithi137 ceph-mon[35787]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-23T08:48:05.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:05 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:05.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:05 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:05.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:05 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:48:05.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:05 smithi059 ceph-mon[28424]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-23T08:48:05.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:05 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:05.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:05 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:05.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:05 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:48:07.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:07 smithi137 ceph-mon[35787]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-23T08:48:07.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:07 smithi059 ceph-mon[28424]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-23T08:48:08.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:08.249+0000 7fd3b3fff700 1 -- 172.21.15.137:0/4183357734 <== mgr.14210 v2:172.21.15.59:6800/3219735162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd3bc05e260 con 0x7fd3a8068f70 2024-01-23T08:48:08.246 INFO:teuthology.orchestra.run.smithi137.stdout:Created osd(s) 4 on host 'smithi137' 2024-01-23T08:48:08.248 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:08.251+0000 7fd3c139c700 1 -- 172.21.15.137:0/4183357734 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fd3a8068f70 msgr2=0x7fd3a806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:08.248 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:08.251+0000 7fd3c139c700 1 --2- 172.21.15.137:0/4183357734 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fd3a8068f70 0x7fd3a806b420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac004f90 tx=0x7fd3ac009e30 comp rx=0 tx=0).stop 2024-01-23T08:48:08.248 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:08.251+0000 7fd3c139c700 1 -- 172.21.15.137:0/4183357734 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3bc0fdfe0 msgr2=0x7fd3bc070170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:08.248 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:08.251+0000 7fd3c139c700 1 --2- 172.21.15.137:0/4183357734 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3bc0fdfe0 0x7fd3bc070170 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400c8c0 tx=0x7fd3a40044c0 comp rx=0 tx=0).stop 2024-01-23T08:48:08.248 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:08.251+0000 7fd3c139c700 1 -- 172.21.15.137:0/4183357734 shutdown_connections 2024-01-23T08:48:08.248 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:08.251+0000 7fd3c139c700 1 --2- 172.21.15.137:0/4183357734 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fd3a8068f70 0x7fd3a806b420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:08.248 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:08.251+0000 7fd3c139c700 1 --2- 172.21.15.137:0/4183357734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3bc0fcdd0 0x7fd3bc071cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:08.248 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:08.251+0000 7fd3c139c700 1 --2- 172.21.15.137:0/4183357734 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3bc0fdfe0 0x7fd3bc070170 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:08.249 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:08.251+0000 7fd3c139c700 1 -- 172.21.15.137:0/4183357734 >> 172.21.15.137:0/4183357734 conn(0x7fd3bc0f84b0 msgr2=0x7fd3bc0fa8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:48:08.249 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:08.251+0000 7fd3c139c700 1 -- 172.21.15.137:0/4183357734 shutdown_connections 2024-01-23T08:48:08.249 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:08.251+0000 7fd3c139c700 1 -- 172.21.15.137:0/4183357734 wait complete. 2024-01-23T08:48:08.851 DEBUG:teuthology.orchestra.run.smithi137:osd.4> sudo journalctl -f -n 0 -u ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@osd.4.service 2024-01-23T08:48:08.853 INFO:tasks.cephadm:Deploying osd.5 on smithi137 with /dev/vg_nvme/lv_2... 2024-01-23T08:48:08.853 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-01-23T08:48:09.161 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi137/config 2024-01-23T08:48:09.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:09 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:09.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:09 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:09.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:09 smithi059 ceph-mon[28424]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-23T08:48:09.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:09 smithi059 ceph-mon[28424]: from='osd.4 [v2:172.21.15.137:6808/708078156,v1:172.21.15.137:6809/708078156]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-23T08:48:09.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:09 smithi059 ceph-mon[28424]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-23T08:48:09.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:09 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:09.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:09 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:09.530 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:09 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:09.530 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:09 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:09.530 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:09 smithi137 ceph-mon[35787]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-23T08:48:09.530 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:09 smithi137 ceph-mon[35787]: from='osd.4 [v2:172.21.15.137:6808/708078156,v1:172.21.15.137:6809/708078156]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-23T08:48:09.530 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:09 smithi137 ceph-mon[35787]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-23T08:48:09.530 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:09 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:09.530 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:09 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:10.217 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:10 smithi137 ceph-mon[35787]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-01-23T08:48:10.217 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:10 smithi137 ceph-mon[35787]: osdmap e33: 5 total, 4 up, 5 in 2024-01-23T08:48:10.217 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:10 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-23T08:48:10.217 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:10 smithi137 ceph-mon[35787]: from='osd.4 [v2:172.21.15.137:6808/708078156,v1:172.21.15.137:6809/708078156]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-01-23T08:48:10.217 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:10 smithi137 ceph-mon[35787]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-01-23T08:48:10.217 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jan 23 08:48:10 smithi137 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-4[47616]: 2024-01-23T08:48:10.213+0000 7f4d2cab3700 -1 osd.4 0 waiting for initial osdmap 2024-01-23T08:48:10.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:10 smithi059 ceph-mon[28424]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-01-23T08:48:10.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:10 smithi059 ceph-mon[28424]: osdmap e33: 5 total, 4 up, 5 in 2024-01-23T08:48:10.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:10 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-23T08:48:10.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:10 smithi059 ceph-mon[28424]: from='osd.4 [v2:172.21.15.137:6808/708078156,v1:172.21.15.137:6809/708078156]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-01-23T08:48:10.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:10 smithi059 ceph-mon[28424]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-01-23T08:48:10.480 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jan 23 08:48:10 smithi137 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-4[47616]: 2024-01-23T08:48:10.220+0000 7f4d258aa700 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-23T08:48:11.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:11 smithi059 ceph-mon[28424]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-01-23T08:48:11.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:11 smithi059 ceph-mon[28424]: osdmap e34: 5 total, 4 up, 5 in 2024-01-23T08:48:11.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:11 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-23T08:48:11.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:11 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-23T08:48:11.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:11 smithi059 ceph-mon[28424]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-23T08:48:11.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:11 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:48:11.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:11 smithi137 ceph-mon[35787]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-01-23T08:48:11.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:11 smithi137 ceph-mon[35787]: osdmap e34: 5 total, 4 up, 5 in 2024-01-23T08:48:11.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:11 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-23T08:48:11.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:11 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-23T08:48:11.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:11 smithi137 ceph-mon[35787]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-23T08:48:11.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:11 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:48:12.067 INFO:teuthology.orchestra.run.smithi137.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-01-23T08:48:12.067 INFO:teuthology.orchestra.run.smithi137.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-01-23T08:48:12.067 INFO:teuthology.orchestra.run.smithi137.stderr: stderr: 10+0 records in 2024-01-23T08:48:12.067 INFO:teuthology.orchestra.run.smithi137.stderr:10+0 records out 2024-01-23T08:48:12.067 INFO:teuthology.orchestra.run.smithi137.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122146 s, 858 MB/s 2024-01-23T08:48:12.067 INFO:teuthology.orchestra.run.smithi137.stderr:--> Zapping successful for: 2024-01-23T08:48:12.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:12 smithi059 ceph-mon[28424]: purged_snaps scrub starts 2024-01-23T08:48:12.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:12 smithi059 ceph-mon[28424]: purged_snaps scrub ok 2024-01-23T08:48:12.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:12 smithi059 ceph-mon[28424]: osd.4 [v2:172.21.15.137:6808/708078156,v1:172.21.15.137:6809/708078156] boot 2024-01-23T08:48:12.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:12 smithi059 ceph-mon[28424]: osdmap e35: 5 total, 5 up, 5 in 2024-01-23T08:48:12.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:12 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-23T08:48:12.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:12 smithi137 ceph-mon[35787]: purged_snaps scrub starts 2024-01-23T08:48:12.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:12 smithi137 ceph-mon[35787]: purged_snaps scrub ok 2024-01-23T08:48:12.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:12 smithi137 ceph-mon[35787]: osd.4 [v2:172.21.15.137:6808/708078156,v1:172.21.15.137:6809/708078156] boot 2024-01-23T08:48:12.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:12 smithi137 ceph-mon[35787]: osdmap e35: 5 total, 5 up, 5 in 2024-01-23T08:48:12.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:12 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-23T08:48:12.737 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi137:vg_nvme/lv_2 2024-01-23T08:48:13.097 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi137/config 2024-01-23T08:48:13.488 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:13 smithi137 ceph-mon[35787]: osdmap e36: 5 total, 5 up, 5 in 2024-01-23T08:48:13.488 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:13 smithi137 ceph-mon[35787]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:13.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:13 smithi059 ceph-mon[28424]: osdmap e36: 5 total, 5 up, 5 in 2024-01-23T08:48:13.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:13 smithi059 ceph-mon[28424]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:14.777 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:14 smithi137 ceph-mon[35787]: Detected new or changed devices on smithi137 2024-01-23T08:48:14.778 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:14 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:14.778 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:14 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:14.778 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:14 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:48:14.778 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:14 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:48:14.778 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:14 smithi137 ceph-mon[35787]: Adjusting osd_memory_target on smithi137 to 7968M 2024-01-23T08:48:14.778 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:14 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:14.778 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:14 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:48:14.778 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:14 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:48:14.778 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:14 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:14.778 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:14 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:48:14.778 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:14 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:48:14.778 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:14 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:48:14.778 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:14 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:14.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:14 smithi059 ceph-mon[28424]: Detected new or changed devices on smithi137 2024-01-23T08:48:14.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:14 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:14.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:14 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:14.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:14 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:48:14.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:14 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:48:14.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:14 smithi059 ceph-mon[28424]: Adjusting osd_memory_target on smithi137 to 7968M 2024-01-23T08:48:14.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:14 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:14.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:14 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:48:14.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:14 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:48:14.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:14 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:14.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:14 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:48:14.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:14 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:48:14.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:14 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:48:14.972 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:14 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:15.120 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.120+0000 7f8976da2700 1 -- 172.21.15.137:0/670162608 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8970066ae0 msgr2=0x7f89700ff7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:15.120 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.120+0000 7f8976da2700 1 --2- 172.21.15.137:0/670162608 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8970066ae0 0x7f89700ff7c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f896c004940 tx=0x7f896c009e30 comp rx=0 tx=0).stop 2024-01-23T08:48:15.120 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.120+0000 7f8976da2700 1 -- 172.21.15.137:0/670162608 shutdown_connections 2024-01-23T08:48:15.120 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.120+0000 7f8976da2700 1 --2- 172.21.15.137:0/670162608 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89700ffd90 0x7f89701021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:15.120 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.120+0000 7f8976da2700 1 --2- 172.21.15.137:0/670162608 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8970066ae0 0x7f89700ff7c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:15.120 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.120+0000 7f8976da2700 1 -- 172.21.15.137:0/670162608 >> 172.21.15.137:0/670162608 conn(0x7f89700f7190 msgr2=0x7f89700f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:48:15.120 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.120+0000 7f8976da2700 1 -- 172.21.15.137:0/670162608 shutdown_connections 2024-01-23T08:48:15.120 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.120+0000 7f8976da2700 1 -- 172.21.15.137:0/670162608 wait complete. 2024-01-23T08:48:15.120 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.121+0000 7f8976da2700 1 Processor -- start 2024-01-23T08:48:15.120 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.121+0000 7f8976da2700 1 -- start start 2024-01-23T08:48:15.120 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.121+0000 7f8976da2700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8970066ae0 0x7f897010d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:15.121 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.121+0000 7f8976da2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89700ffd90 0x7f897010da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:15.121 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.121+0000 7f8976da2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897010e020 con 0x7f89700ffd90 2024-01-23T08:48:15.121 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.121+0000 7f8976da2700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897010e8c0 con 0x7f8970066ae0 2024-01-23T08:48:15.121 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.121+0000 7f8975da0700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8970066ae0 0x7f897010d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:15.121 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.121+0000 7f8975da0700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8970066ae0 0x7f897010d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.137:42774/0 (socket says 172.21.15.137:42774) 2024-01-23T08:48:15.121 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.121+0000 7f8975da0700 1 -- 172.21.15.137:0/1112470756 learned_addr learned my addr 172.21.15.137:0/1112470756 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-23T08:48:15.121 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.121+0000 7f897559f700 1 --2- 172.21.15.137:0/1112470756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89700ffd90 0x7f897010da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:15.121 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.122+0000 7f8975da0700 1 -- 172.21.15.137:0/1112470756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89700ffd90 msgr2=0x7f897010da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:15.121 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.122+0000 7f8975da0700 1 --2- 172.21.15.137:0/1112470756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89700ffd90 0x7f897010da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:15.121 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.122+0000 7f8975da0700 1 -- 172.21.15.137:0/1112470756 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f896c005040 con 0x7f8970066ae0 2024-01-23T08:48:15.121 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.122+0000 7f897559f700 1 --2- 172.21.15.137:0/1112470756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89700ffd90 0x7f897010da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-23T08:48:15.121 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.122+0000 7f8975da0700 1 --2- 172.21.15.137:0/1112470756 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8970066ae0 0x7f897010d4e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f896c009360 tx=0x7f896c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:48:15.121 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.122+0000 7f89667fc700 1 -- 172.21.15.137:0/1112470756 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f896c00e040 con 0x7f8970066ae0 2024-01-23T08:48:15.121 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.122+0000 7f89667fc700 1 -- 172.21.15.137:0/1112470756 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f896c00bcd0 con 0x7f8970066ae0 2024-01-23T08:48:15.121 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.122+0000 7f8976da2700 1 -- 172.21.15.137:0/1112470756 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f897010eb40 con 0x7f8970066ae0 2024-01-23T08:48:15.122 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.123+0000 7f89667fc700 1 -- 172.21.15.137:0/1112470756 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f896c01c630 con 0x7f8970066ae0 2024-01-23T08:48:15.122 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.123+0000 7f8976da2700 1 -- 172.21.15.137:0/1112470756 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f897010f080 con 0x7f8970066ae0 2024-01-23T08:48:15.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.123+0000 7f89667fc700 1 -- 172.21.15.137:0/1112470756 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f896c01c8f0 con 0x7f8970066ae0 2024-01-23T08:48:15.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.123+0000 7f8976da2700 1 -- 172.21.15.137:0/1112470756 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f897004bd60 con 0x7f8970066ae0 2024-01-23T08:48:15.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.124+0000 7f89667fc700 1 --2- 172.21.15.137:0/1112470756 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f895c068f60 0x7f895c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:15.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.124+0000 7f89667fc700 1 -- 172.21.15.137:0/1112470756 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f896c087cf0 con 0x7f8970066ae0 2024-01-23T08:48:15.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.124+0000 7f897559f700 1 --2- 172.21.15.137:0/1112470756 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f895c068f60 0x7f895c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:15.123 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.125+0000 7f897559f700 1 --2- 172.21.15.137:0/1112470756 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f895c068f60 0x7f895c06b410 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8960009fd0 tx=0x7f8960009ee0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:48:15.126 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.128+0000 7f89667fc700 1 -- 172.21.15.137:0/1112470756 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f896c058290 con 0x7f8970066ae0 2024-01-23T08:48:15.322 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:15.324+0000 7f8976da2700 1 -- 172.21.15.137:0/1112470756 --> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f897004bd60 con 0x7f895c068f60 2024-01-23T08:48:15.942 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:15 smithi137 ceph-mon[35787]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:15.942 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:15 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-23T08:48:15.942 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:15 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-23T08:48:15.942 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:15 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:48:15.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:15 smithi059 ceph-mon[28424]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:15.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:15 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-23T08:48:15.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:15 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-23T08:48:15.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:15 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:48:16.771 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:16 smithi137 ceph-mon[35787]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:48:16.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:16 smithi059 ceph-mon[28424]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:48:17.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:17 smithi059 ceph-mon[28424]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:18.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:17 smithi137 ceph-mon[35787]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:18.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:18 smithi059 ceph-mon[28424]: from='client.? 172.21.15.137:0/2313431395' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fdaab2d1-80f0-4928-817e-8ac45394367e"}]: dispatch 2024-01-23T08:48:18.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:18 smithi059 ceph-mon[28424]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fdaab2d1-80f0-4928-817e-8ac45394367e"}]: dispatch 2024-01-23T08:48:18.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:18 smithi059 ceph-mon[28424]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fdaab2d1-80f0-4928-817e-8ac45394367e"}]': finished 2024-01-23T08:48:18.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:18 smithi059 ceph-mon[28424]: osdmap e37: 6 total, 5 up, 6 in 2024-01-23T08:48:18.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:18 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-23T08:48:18.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:18 smithi059 ceph-mon[28424]: from='client.? 172.21.15.137:0/41542666' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-23T08:48:19.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:18 smithi137 ceph-mon[35787]: from='client.? 172.21.15.137:0/2313431395' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fdaab2d1-80f0-4928-817e-8ac45394367e"}]: dispatch 2024-01-23T08:48:19.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:18 smithi137 ceph-mon[35787]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fdaab2d1-80f0-4928-817e-8ac45394367e"}]: dispatch 2024-01-23T08:48:19.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:18 smithi137 ceph-mon[35787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fdaab2d1-80f0-4928-817e-8ac45394367e"}]': finished 2024-01-23T08:48:19.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:18 smithi137 ceph-mon[35787]: osdmap e37: 6 total, 5 up, 6 in 2024-01-23T08:48:19.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:18 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-23T08:48:19.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:18 smithi137 ceph-mon[35787]: from='client.? 172.21.15.137:0/41542666' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-23T08:48:19.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:19 smithi059 ceph-mon[28424]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:20.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:19 smithi137 ceph-mon[35787]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:21.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:21 smithi137 ceph-mon[35787]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:21.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:21 smithi059 ceph-mon[28424]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:23.702 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:23 smithi137 ceph-mon[35787]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:23.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:23 smithi059 ceph-mon[28424]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:25.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:25 smithi059 ceph-mon[28424]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:25.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:25 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-01-23T08:48:25.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:25 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:48:25.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:25 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:48:25.731 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:25 smithi137 ceph-mon[35787]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:25.731 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:25 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-01-23T08:48:25.731 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:25 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:48:25.732 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:25 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:48:26.646 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:26 smithi137 ceph-mon[35787]: Deploying daemon osd.5 on smithi137 2024-01-23T08:48:26.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:26 smithi059 ceph-mon[28424]: Deploying daemon osd.5 on smithi137 2024-01-23T08:48:27.572 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:27 smithi137 ceph-mon[35787]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:27.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:27 smithi059 ceph-mon[28424]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:29.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:29 smithi059 ceph-mon[28424]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:29.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:29 smithi137 ceph-mon[35787]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:31.291 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:31 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:31.291 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:31 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:31.291 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:31 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:48:31.291 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:31 smithi137 ceph-mon[35787]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:31.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:31 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:31.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:31 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:31.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:31 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:48:31.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:31 smithi059 ceph-mon[28424]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:33.437 INFO:teuthology.orchestra.run.smithi137.stdout:Created osd(s) 5 on host 'smithi137' 2024-01-23T08:48:33.437 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:33.439+0000 7f89667fc700 1 -- 172.21.15.137:0/1112470756 <== mgr.14210 v2:172.21.15.59:6800/3219735162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f897004bd60 con 0x7f895c068f60 2024-01-23T08:48:33.439 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:33.441+0000 7f8976da2700 1 -- 172.21.15.137:0/1112470756 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f895c068f60 msgr2=0x7f895c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:33.439 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:33.441+0000 7f8976da2700 1 --2- 172.21.15.137:0/1112470756 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f895c068f60 0x7f895c06b410 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8960009fd0 tx=0x7f8960009ee0 comp rx=0 tx=0).stop 2024-01-23T08:48:33.439 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:33.441+0000 7f8976da2700 1 -- 172.21.15.137:0/1112470756 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8970066ae0 msgr2=0x7f897010d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:33.439 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:33.441+0000 7f8976da2700 1 --2- 172.21.15.137:0/1112470756 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8970066ae0 0x7f897010d4e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f896c009360 tx=0x7f896c009e30 comp rx=0 tx=0).stop 2024-01-23T08:48:33.439 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:33.441+0000 7f8976da2700 1 -- 172.21.15.137:0/1112470756 shutdown_connections 2024-01-23T08:48:33.439 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:33.441+0000 7f8976da2700 1 --2- 172.21.15.137:0/1112470756 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f895c068f60 0x7f895c06b410 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:33.439 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:33.441+0000 7f8976da2700 1 --2- 172.21.15.137:0/1112470756 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8970066ae0 0x7f897010d4e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:33.439 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:33.441+0000 7f8976da2700 1 --2- 172.21.15.137:0/1112470756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89700ffd90 0x7f897010da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:33.439 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:33.441+0000 7f8976da2700 1 -- 172.21.15.137:0/1112470756 >> 172.21.15.137:0/1112470756 conn(0x7f89700f7190 msgr2=0x7f89700f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:48:33.440 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:33.441+0000 7f8976da2700 1 -- 172.21.15.137:0/1112470756 shutdown_connections 2024-01-23T08:48:33.440 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:33.441+0000 7f8976da2700 1 -- 172.21.15.137:0/1112470756 wait complete. 2024-01-23T08:48:33.629 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:33 smithi137 ceph-mon[35787]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:33.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:33 smithi059 ceph-mon[28424]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:34.251 DEBUG:teuthology.orchestra.run.smithi137:osd.5> sudo journalctl -f -n 0 -u ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@osd.5.service 2024-01-23T08:48:34.254 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-01-23T08:48:34.254 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph osd stat -f json 2024-01-23T08:48:34.537 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:48:34.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:34 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:34.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:34 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:34.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:34 smithi137 ceph-mon[35787]: from='osd.5 [v2:172.21.15.137:6816/765377139,v1:172.21.15.137:6817/765377139]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-23T08:48:34.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:34 smithi137 ceph-mon[35787]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-23T08:48:34.660 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:34 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:34.660 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:34 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:34.660 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:34 smithi059 ceph-mon[28424]: from='osd.5 [v2:172.21.15.137:6816/765377139,v1:172.21.15.137:6817/765377139]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-23T08:48:34.660 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:34 smithi059 ceph-mon[28424]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-23T08:48:35.654 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:35 smithi137 ceph-mon[35787]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:35.655 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:35 smithi137 ceph-mon[35787]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-01-23T08:48:35.655 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:35 smithi137 ceph-mon[35787]: osdmap e38: 6 total, 5 up, 6 in 2024-01-23T08:48:35.655 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:35 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-23T08:48:35.655 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:35 smithi137 ceph-mon[35787]: from='osd.5 [v2:172.21.15.137:6816/765377139,v1:172.21.15.137:6817/765377139]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-01-23T08:48:35.655 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:35 smithi137 ceph-mon[35787]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-01-23T08:48:35.655 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:35 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:35.655 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:35 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:35.655 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jan 23 08:48:35 smithi137 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-5[54350]: 2024-01-23T08:48:35.416+0000 7f660ac69700 -1 osd.5 0 waiting for initial osdmap 2024-01-23T08:48:35.655 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jan 23 08:48:35 smithi137 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-5[54350]: 2024-01-23T08:48:35.424+0000 7f6605a64700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-23T08:48:35.667 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:35 smithi059 ceph-mon[28424]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:35.667 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:35 smithi059 ceph-mon[28424]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-01-23T08:48:35.667 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:35 smithi059 ceph-mon[28424]: osdmap e38: 6 total, 5 up, 6 in 2024-01-23T08:48:35.667 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:35 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-23T08:48:35.667 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:35 smithi059 ceph-mon[28424]: from='osd.5 [v2:172.21.15.137:6816/765377139,v1:172.21.15.137:6817/765377139]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-01-23T08:48:35.667 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:35 smithi059 ceph-mon[28424]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-01-23T08:48:35.667 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:35 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:35.667 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:35 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:36.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.132+0000 7f78bad59700 1 -- 172.21.15.59:0/2467307597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78b40fdfe0 msgr2=0x7f78b40fe430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:36.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.132+0000 7f78bad59700 1 --2- 172.21.15.59:0/2467307597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78b40fdfe0 0x7f78b40fe430 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f78a0004940 tx=0x7f78a0009e30 comp rx=0 tx=0).stop 2024-01-23T08:48:36.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.132+0000 7f78bad59700 1 -- 172.21.15.59:0/2467307597 shutdown_connections 2024-01-23T08:48:36.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.132+0000 7f78bad59700 1 --2- 172.21.15.59:0/2467307597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78b40fdfe0 0x7f78b40fe430 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:36.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.132+0000 7f78bad59700 1 --2- 172.21.15.59:0/2467307597 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f78b40fcdd0 0x7f78b40fd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:36.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.132+0000 7f78bad59700 1 -- 172.21.15.59:0/2467307597 >> 172.21.15.59:0/2467307597 conn(0x7f78b40f84b0 msgr2=0x7f78b40fa8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:48:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.132+0000 7f78bad59700 1 -- 172.21.15.59:0/2467307597 shutdown_connections 2024-01-23T08:48:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.132+0000 7f78bad59700 1 -- 172.21.15.59:0/2467307597 wait complete. 2024-01-23T08:48:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.132+0000 7f78bad59700 1 Processor -- start 2024-01-23T08:48:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.132+0000 7f78bad59700 1 -- start start 2024-01-23T08:48:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.133+0000 7f78bad59700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78b40fcdd0 0x7f78b410d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.133+0000 7f78bad59700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f78b40fdfe0 0x7f78b410dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.133+0000 7f78bad59700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b410e2e0 con 0x7f78b40fcdd0 2024-01-23T08:48:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.133+0000 7f78bad59700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b410eb80 con 0x7f78b40fdfe0 2024-01-23T08:48:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.133+0000 7f78b8af5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78b40fcdd0 0x7f78b410d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.133+0000 7f78b8af5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78b40fcdd0 0x7f78b410d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48840/0 (socket says 172.21.15.59:48840) 2024-01-23T08:48:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.133+0000 7f78b8af5700 1 -- 172.21.15.59:0/1017434062 learned_addr learned my addr 172.21.15.59:0/1017434062 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:48:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.133+0000 7f78b3fff700 1 --2- 172.21.15.59:0/1017434062 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f78b40fdfe0 0x7f78b410dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.133+0000 7f78b8af5700 1 -- 172.21.15.59:0/1017434062 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f78b40fdfe0 msgr2=0x7f78b410dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.133+0000 7f78b8af5700 1 --2- 172.21.15.59:0/1017434062 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f78b40fdfe0 0x7f78b410dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.133+0000 7f78b8af5700 1 -- 172.21.15.59:0/1017434062 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a0005040 con 0x7f78b40fcdd0 2024-01-23T08:48:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.133+0000 7f78b8af5700 1 --2- 172.21.15.59:0/1017434062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78b40fcdd0 0x7f78b410d7a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f78a8002a80 tx=0x7f78a8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:48:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.134+0000 7f78b17fa700 1 -- 172.21.15.59:0/1017434062 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78a8007cc0 con 0x7f78b40fcdd0 2024-01-23T08:48:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.134+0000 7f78b17fa700 1 -- 172.21.15.59:0/1017434062 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f78a800f040 con 0x7f78b40fcdd0 2024-01-23T08:48:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.134+0000 7f78bad59700 1 -- 172.21.15.59:0/1017434062 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78b410ee60 con 0x7f78b40fcdd0 2024-01-23T08:48:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.134+0000 7f78b17fa700 1 -- 172.21.15.59:0/1017434062 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78a800d9c0 con 0x7f78b40fcdd0 2024-01-23T08:48:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.134+0000 7f78bad59700 1 -- 172.21.15.59:0/1017434062 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b410f3a0 con 0x7f78b40fcdd0 2024-01-23T08:48:36.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.135+0000 7f78bad59700 1 -- 172.21.15.59:0/1017434062 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78b41076e0 con 0x7f78b40fcdd0 2024-01-23T08:48:36.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.135+0000 7f78b17fa700 1 -- 172.21.15.59:0/1017434062 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f78a800db20 con 0x7f78b40fcdd0 2024-01-23T08:48:36.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.135+0000 7f78b17fa700 1 --2- 172.21.15.59:0/1017434062 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f78a4068f60 0x7f78a406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:36.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.135+0000 7f78b17fa700 1 -- 172.21.15.59:0/1017434062 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4354+0+0 (secure 0 0 0) 0x7f78a8086670 con 0x7f78b40fcdd0 2024-01-23T08:48:36.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.136+0000 7f78b3fff700 1 --2- 172.21.15.59:0/1017434062 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f78a4068f60 0x7f78a406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:36.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.136+0000 7f78b3fff700 1 --2- 172.21.15.59:0/1017434062 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f78a4068f60 0x7f78a406b410 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f78a000b3b0 tx=0x7f78a0009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:48:36.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.140+0000 7f78b17fa700 1 -- 172.21.15.59:0/1017434062 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f78a80569a0 con 0x7f78b40fcdd0 2024-01-23T08:48:36.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.333+0000 7f78bad59700 1 -- 172.21.15.59:0/1017434062 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f78b404bd60 con 0x7f78b40fcdd0 2024-01-23T08:48:36.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.334+0000 7f78b17fa700 1 -- 172.21.15.59:0/1017434062 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v39) v1 ==== 74+0+130 (secure 0 0 0) 0x7f78a8056530 con 0x7f78b40fcdd0 2024-01-23T08:48:36.331 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:48:36.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.336+0000 7f78bad59700 1 -- 172.21.15.59:0/1017434062 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f78a4068f60 msgr2=0x7f78a406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:36.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.336+0000 7f78bad59700 1 --2- 172.21.15.59:0/1017434062 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f78a4068f60 0x7f78a406b410 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f78a000b3b0 tx=0x7f78a0009e30 comp rx=0 tx=0).stop 2024-01-23T08:48:36.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.336+0000 7f78bad59700 1 -- 172.21.15.59:0/1017434062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78b40fcdd0 msgr2=0x7f78b410d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:36.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.336+0000 7f78bad59700 1 --2- 172.21.15.59:0/1017434062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78b40fcdd0 0x7f78b410d7a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f78a8002a80 tx=0x7f78a8004600 comp rx=0 tx=0).stop 2024-01-23T08:48:36.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.336+0000 7f78bad59700 1 -- 172.21.15.59:0/1017434062 shutdown_connections 2024-01-23T08:48:36.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.336+0000 7f78bad59700 1 --2- 172.21.15.59:0/1017434062 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f78a4068f60 0x7f78a406b410 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:36.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.336+0000 7f78bad59700 1 --2- 172.21.15.59:0/1017434062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78b40fcdd0 0x7f78b410d7a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:36.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.336+0000 7f78bad59700 1 --2- 172.21.15.59:0/1017434062 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f78b40fdfe0 0x7f78b410dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:36.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.336+0000 7f78bad59700 1 -- 172.21.15.59:0/1017434062 >> 172.21.15.59:0/1017434062 conn(0x7f78b40f84b0 msgr2=0x7f78b40fa850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:48:36.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.336+0000 7f78bad59700 1 -- 172.21.15.59:0/1017434062 shutdown_connections 2024-01-23T08:48:36.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:36.336+0000 7f78bad59700 1 -- 172.21.15.59:0/1017434062 wait complete. 2024-01-23T08:48:36.497 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:36 smithi059 ceph-mon[28424]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-01-23T08:48:36.497 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:36 smithi059 ceph-mon[28424]: osdmap e39: 6 total, 5 up, 6 in 2024-01-23T08:48:36.497 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:36 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-23T08:48:36.497 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:36 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-23T08:48:36.497 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:36 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/1017434062' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-23T08:48:36.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:36 smithi137 ceph-mon[35787]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-01-23T08:48:36.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:36 smithi137 ceph-mon[35787]: osdmap e39: 6 total, 5 up, 6 in 2024-01-23T08:48:36.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:36 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-23T08:48:36.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:36 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-23T08:48:36.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:36 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/1017434062' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-23T08:48:37.106 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":39,"num_osds":6,"num_up_osds":5,"osd_up_since":1705999691,"num_in_osds":6,"osd_in_since":1705999697,"num_remapped_pgs":0} 2024-01-23T08:48:37.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:37 smithi137 ceph-mon[35787]: purged_snaps scrub starts 2024-01-23T08:48:37.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:37 smithi137 ceph-mon[35787]: purged_snaps scrub ok 2024-01-23T08:48:37.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:37 smithi137 ceph-mon[35787]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:37.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:37 smithi137 ceph-mon[35787]: osd.5 [v2:172.21.15.137:6816/765377139,v1:172.21.15.137:6817/765377139] boot 2024-01-23T08:48:37.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:37 smithi137 ceph-mon[35787]: osdmap e40: 6 total, 6 up, 6 in 2024-01-23T08:48:37.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:37 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-23T08:48:37.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:37 smithi059 ceph-mon[28424]: purged_snaps scrub starts 2024-01-23T08:48:37.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:37 smithi059 ceph-mon[28424]: purged_snaps scrub ok 2024-01-23T08:48:37.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:37 smithi059 ceph-mon[28424]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-23T08:48:37.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:37 smithi059 ceph-mon[28424]: osd.5 [v2:172.21.15.137:6816/765377139,v1:172.21.15.137:6817/765377139] boot 2024-01-23T08:48:37.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:37 smithi059 ceph-mon[28424]: osdmap e40: 6 total, 6 up, 6 in 2024-01-23T08:48:37.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:37 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-23T08:48:38.108 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph osd stat -f json 2024-01-23T08:48:38.380 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:48:38.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:38 smithi059 ceph-mon[28424]: osdmap e41: 6 total, 6 up, 6 in 2024-01-23T08:48:38.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:38 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:38.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:38 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:38.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:38 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:48:38.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:38 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:48:38.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:38 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:48:38.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:38 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:38.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:38 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:48:38.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:38 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:48:38.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:38 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:38.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:38 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:48:38.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:38 smithi137 ceph-mon[35787]: osdmap e41: 6 total, 6 up, 6 in 2024-01-23T08:48:38.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:38 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:38.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:38 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:38.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:38 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:48:38.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:38 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:48:38.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:38 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-01-23T08:48:38.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:38 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:38.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:38 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:48:38.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:38 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:48:38.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:38 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:38.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:38 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:48:39.512 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:39 smithi059 ceph-mon[28424]: Detected new or changed devices on smithi137 2024-01-23T08:48:39.512 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:39 smithi059 ceph-mon[28424]: Adjusting osd_memory_target on smithi137 to 5312M 2024-01-23T08:48:39.512 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:39 smithi059 ceph-mon[28424]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:48:39.512 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:39 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:48:39.512 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:39 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:48:39.512 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:39 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:39.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:39 smithi137 ceph-mon[35787]: Detected new or changed devices on smithi137 2024-01-23T08:48:39.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:39 smithi137 ceph-mon[35787]: Adjusting osd_memory_target on smithi137 to 5312M 2024-01-23T08:48:39.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:39 smithi137 ceph-mon[35787]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:48:39.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:39 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:48:39.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:39 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:48:39.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:39 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:48:40.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.062+0000 7f1678d02700 1 -- 172.21.15.59:0/2507223210 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16740fbf30 msgr2=0x7f16740fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:40.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.062+0000 7f1678d02700 1 --2- 172.21.15.59:0/2507223210 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16740fbf30 0x7f16740fc3a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1668004940 tx=0x7f1668009e30 comp rx=0 tx=0).stop 2024-01-23T08:48:40.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.063+0000 7f1678d02700 1 -- 172.21.15.59:0/2507223210 shutdown_connections 2024-01-23T08:48:40.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.063+0000 7f1678d02700 1 --2- 172.21.15.59:0/2507223210 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16740fbf30 0x7f16740fc3a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:40.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.063+0000 7f1678d02700 1 --2- 172.21.15.59:0/2507223210 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16740fd2c0 0x7f16740fd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:40.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.063+0000 7f1678d02700 1 -- 172.21.15.59:0/2507223210 >> 172.21.15.59:0/2507223210 conn(0x7f16740f6b30 msgr2=0x7f16740f8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:48:40.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.063+0000 7f1678d02700 1 -- 172.21.15.59:0/2507223210 shutdown_connections 2024-01-23T08:48:40.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.063+0000 7f1678d02700 1 -- 172.21.15.59:0/2507223210 wait complete. 2024-01-23T08:48:40.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.063+0000 7f1678d02700 1 Processor -- start 2024-01-23T08:48:40.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.063+0000 7f1678d02700 1 -- start start 2024-01-23T08:48:40.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.063+0000 7f1678d02700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16740fbf30 0x7f167410d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:40.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.063+0000 7f1678d02700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16740fd2c0 0x7f167410dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:40.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.063+0000 7f1678d02700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167410e390 con 0x7f16740fbf30 2024-01-23T08:48:40.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.063+0000 7f1678d02700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167410ec30 con 0x7f16740fd2c0 2024-01-23T08:48:40.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.063+0000 7f167259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16740fbf30 0x7f167410d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:40.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.063+0000 7f167259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16740fbf30 0x7f167410d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39882/0 (socket says 172.21.15.59:39882) 2024-01-23T08:48:40.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.063+0000 7f167259c700 1 -- 172.21.15.59:0/2352046310 learned_addr learned my addr 172.21.15.59:0/2352046310 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:48:40.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.064+0000 7f1671d9b700 1 --2- 172.21.15.59:0/2352046310 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16740fd2c0 0x7f167410dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:40.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.064+0000 7f167259c700 1 -- 172.21.15.59:0/2352046310 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16740fd2c0 msgr2=0x7f167410dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:40.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.064+0000 7f167259c700 1 --2- 172.21.15.59:0/2352046310 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16740fd2c0 0x7f167410dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:40.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.064+0000 7f167259c700 1 -- 172.21.15.59:0/2352046310 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1668005040 con 0x7f16740fbf30 2024-01-23T08:48:40.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.064+0000 7f167259c700 1 --2- 172.21.15.59:0/2352046310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16740fbf30 0x7f167410d850 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1664002a80 tx=0x7f1664004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:48:40.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.064+0000 7f1662ffd700 1 -- 172.21.15.59:0/2352046310 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1664007cc0 con 0x7f16740fbf30 2024-01-23T08:48:40.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.064+0000 7f1662ffd700 1 -- 172.21.15.59:0/2352046310 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f166400f040 con 0x7f16740fbf30 2024-01-23T08:48:40.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.064+0000 7f1678d02700 1 -- 172.21.15.59:0/2352046310 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f167410ef10 con 0x7f16740fbf30 2024-01-23T08:48:40.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.064+0000 7f1662ffd700 1 -- 172.21.15.59:0/2352046310 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f166400d9c0 con 0x7f16740fbf30 2024-01-23T08:48:40.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.064+0000 7f1678d02700 1 -- 172.21.15.59:0/2352046310 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f167410f450 con 0x7f16740fbf30 2024-01-23T08:48:40.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.065+0000 7f1678d02700 1 -- 172.21.15.59:0/2352046310 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f167404bd60 con 0x7f16740fbf30 2024-01-23T08:48:40.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.066+0000 7f1662ffd700 1 -- 172.21.15.59:0/2352046310 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f166400db20 con 0x7f16740fbf30 2024-01-23T08:48:40.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.066+0000 7f1662ffd700 1 --2- 172.21.15.59:0/2352046310 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f165c068f70 0x7f165c06b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:40.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.066+0000 7f1662ffd700 1 -- 172.21.15.59:0/2352046310 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f16640866b0 con 0x7f16740fbf30 2024-01-23T08:48:40.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.066+0000 7f1671d9b700 1 --2- 172.21.15.59:0/2352046310 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f165c068f70 0x7f165c06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:40.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.066+0000 7f1671d9b700 1 --2- 172.21.15.59:0/2352046310 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f165c068f70 0x7f165c06b420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f166800b3b0 tx=0x7f1668009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:48:40.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.070+0000 7f1662ffd700 1 -- 172.21.15.59:0/2352046310 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1664056950 con 0x7f16740fbf30 2024-01-23T08:48:40.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.261+0000 7f1678d02700 1 -- 172.21.15.59:0/2352046310 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f167410f880 con 0x7f16740fbf30 2024-01-23T08:48:40.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.262+0000 7f1662ffd700 1 -- 172.21.15.59:0/2352046310 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v41) v1 ==== 74+0+130 (secure 0 0 0) 0x7f16640564e0 con 0x7f16740fbf30 2024-01-23T08:48:40.259 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:48:40.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.264+0000 7f1678d02700 1 -- 172.21.15.59:0/2352046310 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f165c068f70 msgr2=0x7f165c06b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:40.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.264+0000 7f1678d02700 1 --2- 172.21.15.59:0/2352046310 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f165c068f70 0x7f165c06b420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f166800b3b0 tx=0x7f1668009e30 comp rx=0 tx=0).stop 2024-01-23T08:48:40.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.264+0000 7f1678d02700 1 -- 172.21.15.59:0/2352046310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16740fbf30 msgr2=0x7f167410d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:40.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.264+0000 7f1678d02700 1 --2- 172.21.15.59:0/2352046310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16740fbf30 0x7f167410d850 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1664002a80 tx=0x7f1664004600 comp rx=0 tx=0).stop 2024-01-23T08:48:40.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.264+0000 7f1678d02700 1 -- 172.21.15.59:0/2352046310 shutdown_connections 2024-01-23T08:48:40.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.264+0000 7f1678d02700 1 --2- 172.21.15.59:0/2352046310 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f165c068f70 0x7f165c06b420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:40.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.264+0000 7f1678d02700 1 --2- 172.21.15.59:0/2352046310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16740fbf30 0x7f167410d850 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:40.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.264+0000 7f1678d02700 1 --2- 172.21.15.59:0/2352046310 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16740fd2c0 0x7f167410dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:40.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.264+0000 7f1678d02700 1 -- 172.21.15.59:0/2352046310 >> 172.21.15.59:0/2352046310 conn(0x7f16740f6b30 msgr2=0x7f16740fae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:48:40.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.264+0000 7f1678d02700 1 -- 172.21.15.59:0/2352046310 shutdown_connections 2024-01-23T08:48:40.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:40.264+0000 7f1678d02700 1 -- 172.21.15.59:0/2352046310 wait complete. 2024-01-23T08:48:40.914 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":41,"num_osds":6,"num_up_osds":6,"osd_up_since":1705999716,"num_in_osds":6,"osd_in_since":1705999697,"num_remapped_pgs":0} 2024-01-23T08:48:40.914 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph osd dump --format=json 2024-01-23T08:48:41.176 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:48:41.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:41 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/2352046310' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-23T08:48:41.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:41 smithi059 ceph-mon[28424]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:48:41.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:41 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:48:41.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:41 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/2352046310' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-23T08:48:41.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:41 smithi137 ceph-mon[35787]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:48:41.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:41 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:48:42.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.792+0000 7fda73a7a700 1 -- 172.21.15.59:0/816751544 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda6c102670 msgr2=0x7fda6c0fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:42.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.792+0000 7fda73a7a700 1 --2- 172.21.15.59:0/816751544 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda6c102670 0x7fda6c0fbd30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fda68004940 tx=0x7fda68009e30 comp rx=0 tx=0).stop 2024-01-23T08:48:42.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.792+0000 7fda73a7a700 1 -- 172.21.15.59:0/816751544 shutdown_connections 2024-01-23T08:48:42.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.792+0000 7fda73a7a700 1 --2- 172.21.15.59:0/816751544 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda6c102670 0x7fda6c0fbd30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:42.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.792+0000 7fda73a7a700 1 --2- 172.21.15.59:0/816751544 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda6c101cd0 0x7fda6c1020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:42.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.792+0000 7fda73a7a700 1 -- 172.21.15.59:0/816751544 >> 172.21.15.59:0/816751544 conn(0x7fda6c071c20 msgr2=0x7fda6c072020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:48:42.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.792+0000 7fda73a7a700 1 -- 172.21.15.59:0/816751544 shutdown_connections 2024-01-23T08:48:42.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.792+0000 7fda73a7a700 1 -- 172.21.15.59:0/816751544 wait complete. 2024-01-23T08:48:42.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.793+0000 7fda73a7a700 1 Processor -- start 2024-01-23T08:48:42.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.793+0000 7fda73a7a700 1 -- start start 2024-01-23T08:48:42.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.793+0000 7fda73a7a700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda6c101cd0 0x7fda6c1138d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:42.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.793+0000 7fda73a7a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda6c102670 0x7fda6c113e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:42.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.793+0000 7fda73a7a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c114460 con 0x7fda6c102670 2024-01-23T08:48:42.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.793+0000 7fda73a7a700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c10d790 con 0x7fda6c101cd0 2024-01-23T08:48:42.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.793+0000 7fda71015700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda6c102670 0x7fda6c113e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:42.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.793+0000 7fda71015700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda6c102670 0x7fda6c113e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39902/0 (socket says 172.21.15.59:39902) 2024-01-23T08:48:42.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.793+0000 7fda71015700 1 -- 172.21.15.59:0/1625565579 learned_addr learned my addr 172.21.15.59:0/1625565579 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:48:42.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.793+0000 7fda71816700 1 --2- 172.21.15.59:0/1625565579 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda6c101cd0 0x7fda6c1138d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:42.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.793+0000 7fda71015700 1 -- 172.21.15.59:0/1625565579 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda6c101cd0 msgr2=0x7fda6c1138d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:42.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.793+0000 7fda71015700 1 --2- 172.21.15.59:0/1625565579 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda6c101cd0 0x7fda6c1138d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:42.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.793+0000 7fda71015700 1 -- 172.21.15.59:0/1625565579 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda68005040 con 0x7fda6c102670 2024-01-23T08:48:42.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.793+0000 7fda71015700 1 --2- 172.21.15.59:0/1625565579 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda6c102670 0x7fda6c113e10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fda68009fd0 tx=0x7fda68009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:48:42.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.794+0000 7fda627fc700 1 -- 172.21.15.59:0/1625565579 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda6800e040 con 0x7fda6c102670 2024-01-23T08:48:42.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.794+0000 7fda73a7a700 1 -- 172.21.15.59:0/1625565579 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda6c10da10 con 0x7fda6c102670 2024-01-23T08:48:42.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.794+0000 7fda627fc700 1 -- 172.21.15.59:0/1625565579 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda6800bc80 con 0x7fda6c102670 2024-01-23T08:48:42.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.794+0000 7fda627fc700 1 -- 172.21.15.59:0/1625565579 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda68013660 con 0x7fda6c102670 2024-01-23T08:48:42.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.794+0000 7fda73a7a700 1 -- 172.21.15.59:0/1625565579 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda6c10df50 con 0x7fda6c102670 2024-01-23T08:48:42.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.795+0000 7fda73a7a700 1 -- 172.21.15.59:0/1625565579 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda6c073740 con 0x7fda6c102670 2024-01-23T08:48:42.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.795+0000 7fda627fc700 1 -- 172.21.15.59:0/1625565579 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fda68013920 con 0x7fda6c102670 2024-01-23T08:48:42.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.795+0000 7fda627fc700 1 --2- 172.21.15.59:0/1625565579 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fda58068f60 0x7fda5806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:42.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.796+0000 7fda627fc700 1 -- 172.21.15.59:0/1625565579 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fda6808bb10 con 0x7fda6c102670 2024-01-23T08:48:42.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.796+0000 7fda71816700 1 --2- 172.21.15.59:0/1625565579 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fda58068f60 0x7fda5806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:42.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.796+0000 7fda71816700 1 --2- 172.21.15.59:0/1625565579 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fda58068f60 0x7fda5806b410 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fda5c007fd0 tx=0x7fda5c009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:48:42.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.800+0000 7fda627fc700 1 -- 172.21.15.59:0/1625565579 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fda6805bd10 con 0x7fda6c102670 2024-01-23T08:48:42.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.992+0000 7fda73a7a700 1 -- 172.21.15.59:0/1625565579 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda6c04bd60 con 0x7fda6c102670 2024-01-23T08:48:42.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.993+0000 7fda627fc700 1 -- 172.21.15.59:0/1625565579 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11069 (secure 0 0 0) 0x7fda6805b8a0 con 0x7fda6c102670 2024-01-23T08:48:42.990 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:48:42.990 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":41,"fsid":"3fc07a38-b9cb-11ee-95b1-87774f69a715","created":"2024-01-23T08:43:05.316045+0000","modified":"2024-01-23T08:48:37.419823+0000","last_up_change":"2024-01-23T08:48:36.414344+0000","last_in_change":"2024-01-23T08:48:17.747223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-23T08:47:24.641141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"444a9c2f-e580-443e-9dfe-36843f8c2e13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2564662034},{"type":"v1","addr":"172.21.15.59:6803","nonce":2564662034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2564662034},{"type":"v1","addr":"172.21.15.59:6805","nonce":2564662034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2564662034},{"type":"v1","addr":"172.21.15.59:6809","nonce":2564662034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2564662034},{"type":"v1","addr":"172.21.15.59:6807","nonce":2564662034}]},"public_addr":"172.21.15.59:6803/2564662034","cluster_addr":"172.21.15.59:6805/2564662034","heartbeat_back_addr":"172.21.15.59:6809/2564662034","heartbeat_front_addr":"172.21.15.59:6807/2564662034","state":["exists","up"]},{"osd":1,"uuid":"c28ec1b9-2f5f-48e6-8322-7092bbe8cdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1673825133},{"type":"v1","addr":"172.21.15.59:6811","nonce":1673825133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1673825133},{"type":"v1","addr":"172.21.15.59:6813","nonce":1673825133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1673825133},{"type":"v1","addr":"172.21.15.59:6817","nonce":1673825133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1673825133},{"type":"v1","addr":"172.21.15.59:6815","nonce":1673825133}]},"public_addr":"172.21.15.59:6811/1673825133","cluster_addr":"172.21.15.59:6813/1673825133","heartbeat_back_addr":"172.21.15.59:6817/1673825133","heartbeat_front_addr":"172.21.15.59:6815/1673825133","state":["exists","up"]},{"osd":2,"uuid":"94d9e85d-6f22-4e83-9f4c-b326e9375e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2143964017},{"type":"v1","addr":"172.21.15.59:6819","nonce":2143964017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2143964017},{"type":"v1","addr":"172.21.15.59:6821","nonce":2143964017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2143964017},{"type":"v1","addr":"172.21.15.59:6825","nonce":2143964017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2143964017},{"type":"v1","addr":"172.21.15.59:6823","nonce":2143964017}]},"public_addr":"172.21.15.59:6819/2143964017","cluster_addr":"172.21.15.59:6821/2143964017","heartbeat_back_addr":"172.21.15.59:6825/2143964017","heartbeat_front_addr":"172.21.15.59:6823/2143964017","state":["exists","up"]},{"osd":3,"uuid":"19dfe941-4ce2-4490-8a09-bf85493593bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2062812031},{"type":"v1","addr":"172.21.15.137:6801","nonce":2062812031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2062812031},{"type":"v1","addr":"172.21.15.137:6803","nonce":2062812031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2062812031},{"type":"v1","addr":"172.21.15.137:6807","nonce":2062812031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2062812031},{"type":"v1","addr":"172.21.15.137:6805","nonce":2062812031}]},"public_addr":"172.21.15.137:6801/2062812031","cluster_addr":"172.21.15.137:6803/2062812031","heartbeat_back_addr":"172.21.15.137:6807/2062812031","heartbeat_front_addr":"172.21.15.137:6805/2062812031","state":["exists","up"]},{"osd":4,"uuid":"a1d6209f-095c-42e2-966a-11559d090531","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":708078156},{"type":"v1","addr":"172.21.15.137:6809","nonce":708078156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":708078156},{"type":"v1","addr":"172.21.15.137:6811","nonce":708078156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":708078156},{"type":"v1","addr":"172.21.15.137:6815","nonce":708078156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":708078156},{"type":"v1","addr":"172.21.15.137:6813","nonce":708078156}]},"public_addr":"172.21.15.137:6809/708078156","cluster_addr":"172.21.15.137:6811/708078156","heartbeat_back_addr":"172.21.15.137:6815/708078156","heartbeat_front_addr":"172.21.15.137:6813/708078156","state":["exists","up"]},{"osd":5,"uuid":"fdaab2d1-80f0-4928-817e-8ac45394367e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":765377139},{"type":"v1","addr":"172.21.15.137:6817","nonce":765377139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":765377139},{"type":"v1","addr":"172.21.15.137:6819","nonce":765377139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":765377139},{"type":"v1","addr":"172.21.15.137:6823","nonce":765377139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":765377139},{"type":"v1","addr":"172.21.15.137:6821","nonce":765377139}]},"public_addr":"172.21.15.137:6817/765377139","cluster_addr":"172.21.15.137:6819/765377139","heartbeat_back_addr":"172.21.15.137:6823/765377139","heartbeat_front_addr":"172.21.15.137:6821/765377139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-23T08:46:25.064905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-23T08:46:52.940650+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-23T08:47:21.360904+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-23T08:47:43.875916+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-23T08:48:09.457117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-23T08:48:34.887400+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:0/2423485094":"2024-01-24T08:45:10.300954+0000","172.21.15.59:0/1598234521":"2024-01-24T08:45:10.300954+0000","172.21.15.59:6801/3614124404":"2024-01-24T08:45:10.300954+0000","172.21.15.59:6800/3614124404":"2024-01-24T08:45:10.300954+0000","172.21.15.59:0/2596349924":"2024-01-24T08:44:03.688872+0000","172.21.15.59:0/3153116091":"2024-01-24T08:44:03.688872+0000","172.21.15.59:6801/1337467262":"2024-01-24T08:44:03.688872+0000","172.21.15.59:6800/1337467262":"2024-01-24T08:44:03.688872+0000","172.21.15.59:0/1368475691":"2024-01-24T08:43:31.660943+0000","172.21.15.59:0/1345586333":"2024-01-24T08:43:31.660943+0000","172.21.15.59:6801/504555577":"2024-01-24T08:43:31.660943+0000","172.21.15.59:0/252476549":"2024-01-24T08:43:31.660943+0000","172.21.15.59:0/2405882784":"2024-01-24T08:45:10.300954+0000","172.21.15.59:6800/504555577":"2024-01-24T08:43:31.660943+0000","172.21.15.59:0/715244053":"2024-01-24T08:44:03.688872+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-23T08:48:42.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.995+0000 7fda73a7a700 1 -- 172.21.15.59:0/1625565579 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fda58068f60 msgr2=0x7fda5806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:42.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.995+0000 7fda73a7a700 1 --2- 172.21.15.59:0/1625565579 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fda58068f60 0x7fda5806b410 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fda5c007fd0 tx=0x7fda5c009040 comp rx=0 tx=0).stop 2024-01-23T08:48:42.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.995+0000 7fda73a7a700 1 -- 172.21.15.59:0/1625565579 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda6c102670 msgr2=0x7fda6c113e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:42.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.995+0000 7fda73a7a700 1 --2- 172.21.15.59:0/1625565579 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda6c102670 0x7fda6c113e10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fda68009fd0 tx=0x7fda68009e30 comp rx=0 tx=0).stop 2024-01-23T08:48:42.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.995+0000 7fda73a7a700 1 -- 172.21.15.59:0/1625565579 shutdown_connections 2024-01-23T08:48:42.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.995+0000 7fda73a7a700 1 --2- 172.21.15.59:0/1625565579 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fda58068f60 0x7fda5806b410 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:42.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.995+0000 7fda73a7a700 1 --2- 172.21.15.59:0/1625565579 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda6c101cd0 0x7fda6c1138d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:42.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.995+0000 7fda73a7a700 1 --2- 172.21.15.59:0/1625565579 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda6c102670 0x7fda6c113e10 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:42.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.995+0000 7fda73a7a700 1 -- 172.21.15.59:0/1625565579 >> 172.21.15.59:0/1625565579 conn(0x7fda6c071c20 msgr2=0x7fda6c0656b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:48:42.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.995+0000 7fda73a7a700 1 -- 172.21.15.59:0/1625565579 shutdown_connections 2024-01-23T08:48:42.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:42.995+0000 7fda73a7a700 1 -- 172.21.15.59:0/1625565579 wait complete. 2024-01-23T08:48:43.652 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:43 smithi059 ceph-mon[28424]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:48:43.652 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:43 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/1625565579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-23T08:48:43.653 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-23T08:47:24.641141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-23T08:48:43.654 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-01-23T08:48:43.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:43 smithi137 ceph-mon[35787]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:48:43.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:43 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/1625565579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-23T08:48:43.914 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:48:45.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.513+0000 7f5e7bf23700 1 -- 172.21.15.59:0/1471503122 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e740fe770 msgr2=0x7f5e74106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:45.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.513+0000 7f5e7bf23700 1 --2- 172.21.15.59:0/1471503122 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e740fe770 0x7f5e74106e40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5e64004940 tx=0x7f5e64009e30 comp rx=0 tx=0).stop 2024-01-23T08:48:45.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.513+0000 7f5e7bf23700 1 -- 172.21.15.59:0/1471503122 shutdown_connections 2024-01-23T08:48:45.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.513+0000 7f5e7bf23700 1 --2- 172.21.15.59:0/1471503122 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e740fe770 0x7f5e74106e40 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:45.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.513+0000 7f5e7bf23700 1 --2- 172.21.15.59:0/1471503122 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e740fddd0 0x7f5e740fe1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:45.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.513+0000 7f5e7bf23700 1 -- 172.21.15.59:0/1471503122 >> 172.21.15.59:0/1471503122 conn(0x7f5e740f7850 msgr2=0x7f5e740f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:48:45.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.513+0000 7f5e7bf23700 1 -- 172.21.15.59:0/1471503122 shutdown_connections 2024-01-23T08:48:45.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.513+0000 7f5e7bf23700 1 -- 172.21.15.59:0/1471503122 wait complete. 2024-01-23T08:48:45.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.514+0000 7f5e7bf23700 1 Processor -- start 2024-01-23T08:48:45.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.514+0000 7f5e7bf23700 1 -- start start 2024-01-23T08:48:45.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.514+0000 7f5e7bf23700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e740fddd0 0x7f5e74113820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:45.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.514+0000 7f5e7bf23700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e740fe770 0x7f5e74113d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:45.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.514+0000 7f5e7bf23700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e741143b0 con 0x7f5e740fe770 2024-01-23T08:48:45.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.514+0000 7f5e7bf23700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7410d6e0 con 0x7f5e740fddd0 2024-01-23T08:48:45.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.514+0000 7f5e794be700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e740fe770 0x7f5e74113d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:45.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.514+0000 7f5e794be700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e740fe770 0x7f5e74113d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39910/0 (socket says 172.21.15.59:39910) 2024-01-23T08:48:45.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.514+0000 7f5e794be700 1 -- 172.21.15.59:0/1328336691 learned_addr learned my addr 172.21.15.59:0/1328336691 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:48:45.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.514+0000 7f5e79cbf700 1 --2- 172.21.15.59:0/1328336691 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e740fddd0 0x7f5e74113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:45.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.514+0000 7f5e794be700 1 -- 172.21.15.59:0/1328336691 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e740fddd0 msgr2=0x7f5e74113820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:45.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.514+0000 7f5e794be700 1 --2- 172.21.15.59:0/1328336691 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e740fddd0 0x7f5e74113820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:45.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.514+0000 7f5e794be700 1 -- 172.21.15.59:0/1328336691 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e64005040 con 0x7f5e740fe770 2024-01-23T08:48:45.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.515+0000 7f5e794be700 1 --2- 172.21.15.59:0/1328336691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e740fe770 0x7f5e74113d60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5e64000c00 tx=0x7f5e64009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:48:45.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.515+0000 7f5e6a7fc700 1 -- 172.21.15.59:0/1328336691 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e64021070 con 0x7f5e740fe770 2024-01-23T08:48:45.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.515+0000 7f5e6a7fc700 1 -- 172.21.15.59:0/1328336691 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e640093b0 con 0x7f5e740fe770 2024-01-23T08:48:45.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.515+0000 7f5e6a7fc700 1 -- 172.21.15.59:0/1328336691 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e6400e070 con 0x7f5e740fe770 2024-01-23T08:48:45.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.515+0000 7f5e7bf23700 1 -- 172.21.15.59:0/1328336691 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e7410d960 con 0x7f5e740fe770 2024-01-23T08:48:45.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.515+0000 7f5e7bf23700 1 -- 172.21.15.59:0/1328336691 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e7410dea0 con 0x7f5e740fe770 2024-01-23T08:48:45.514 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:45 smithi059 ceph-mon[28424]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:48:45.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.516+0000 7f5e7bf23700 1 -- 172.21.15.59:0/1328336691 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e741044a0 con 0x7f5e740fe770 2024-01-23T08:48:45.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.516+0000 7f5e6a7fc700 1 -- 172.21.15.59:0/1328336691 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f5e64013a50 con 0x7f5e740fe770 2024-01-23T08:48:45.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.517+0000 7f5e6a7fc700 1 --2- 172.21.15.59:0/1328336691 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f5e60068f70 0x7f5e6006b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:45.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.517+0000 7f5e6a7fc700 1 -- 172.21.15.59:0/1328336691 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5e64087ac0 con 0x7f5e740fe770 2024-01-23T08:48:45.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.517+0000 7f5e79cbf700 1 --2- 172.21.15.59:0/1328336691 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f5e60068f70 0x7f5e6006b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:45.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.517+0000 7f5e79cbf700 1 --2- 172.21.15.59:0/1328336691 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f5e60068f70 0x7f5e6006b420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5e70009b80 tx=0x7f5e7000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:48:45.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.522+0000 7f5e6a7fc700 1 -- 172.21.15.59:0/1328336691 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5e64057e00 con 0x7f5e740fe770 2024-01-23T08:48:45.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.712+0000 7f5e7bf23700 1 -- 172.21.15.59:0/1328336691 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5e740fb470 con 0x7f5e740fe770 2024-01-23T08:48:45.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.712+0000 7f5e6a7fc700 1 -- 172.21.15.59:0/1328336691 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5e64057990 con 0x7f5e740fe770 2024-01-23T08:48:45.709 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-01-23T08:48:45.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.714+0000 7f5e7bf23700 1 -- 172.21.15.59:0/1328336691 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f5e60068f70 msgr2=0x7f5e6006b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:45.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.714+0000 7f5e7bf23700 1 --2- 172.21.15.59:0/1328336691 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f5e60068f70 0x7f5e6006b420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5e70009b80 tx=0x7f5e7000b040 comp rx=0 tx=0).stop 2024-01-23T08:48:45.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.714+0000 7f5e7bf23700 1 -- 172.21.15.59:0/1328336691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e740fe770 msgr2=0x7f5e74113d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:45.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.714+0000 7f5e7bf23700 1 --2- 172.21.15.59:0/1328336691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e740fe770 0x7f5e74113d60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5e64000c00 tx=0x7f5e64009e30 comp rx=0 tx=0).stop 2024-01-23T08:48:45.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.714+0000 7f5e7bf23700 1 -- 172.21.15.59:0/1328336691 shutdown_connections 2024-01-23T08:48:45.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.714+0000 7f5e7bf23700 1 --2- 172.21.15.59:0/1328336691 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f5e60068f70 0x7f5e6006b420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:45.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.714+0000 7f5e7bf23700 1 --2- 172.21.15.59:0/1328336691 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e740fddd0 0x7f5e74113820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:45.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.714+0000 7f5e7bf23700 1 --2- 172.21.15.59:0/1328336691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e740fe770 0x7f5e74113d60 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:45.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.714+0000 7f5e7bf23700 1 -- 172.21.15.59:0/1328336691 >> 172.21.15.59:0/1328336691 conn(0x7f5e740f7850 msgr2=0x7f5e74101620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:48:45.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.714+0000 7f5e7bf23700 1 -- 172.21.15.59:0/1328336691 shutdown_connections 2024-01-23T08:48:45.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:45.714+0000 7f5e7bf23700 1 -- 172.21.15.59:0/1328336691 wait complete. 2024-01-23T08:48:45.790 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:45 smithi137 ceph-mon[35787]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:48:46.368 INFO:tasks.cephadm:Setting up client nodes... 2024-01-23T08:48:46.368 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-01-23T08:48:46.625 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:48:46.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:46 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/1328336691' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-23T08:48:46.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:46 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/1328336691' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-23T08:48:47.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:47 smithi059 ceph-mon[28424]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:48:47.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:47 smithi137 ceph-mon[35787]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:48:48.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.317+0000 7fcc1cd9b700 1 -- 172.21.15.59:0/4106935569 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc180fe140 msgr2=0x7fcc180fe590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:48.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.317+0000 7fcc1cd9b700 1 --2- 172.21.15.59:0/4106935569 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc180fe140 0x7fcc180fe590 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c004940 tx=0x7fcc0c009e30 comp rx=0 tx=0).stop 2024-01-23T08:48:48.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.317+0000 7fcc1cd9b700 1 -- 172.21.15.59:0/4106935569 shutdown_connections 2024-01-23T08:48:48.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.317+0000 7fcc1cd9b700 1 --2- 172.21.15.59:0/4106935569 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc180fe140 0x7fcc180fe590 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:48.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.317+0000 7fcc1cd9b700 1 --2- 172.21.15.59:0/4106935569 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc180fb0b0 0x7fcc180fb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:48.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.317+0000 7fcc1cd9b700 1 -- 172.21.15.59:0/4106935569 >> 172.21.15.59:0/4106935569 conn(0x7fcc180f6b30 msgr2=0x7fcc180f8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:48:48.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.317+0000 7fcc1cd9b700 1 -- 172.21.15.59:0/4106935569 shutdown_connections 2024-01-23T08:48:48.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.317+0000 7fcc1cd9b700 1 -- 172.21.15.59:0/4106935569 wait complete. 2024-01-23T08:48:48.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.317+0000 7fcc1cd9b700 1 Processor -- start 2024-01-23T08:48:48.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.317+0000 7fcc1cd9b700 1 -- start start 2024-01-23T08:48:48.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.317+0000 7fcc1cd9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc180fb0b0 0x7fcc180fdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:48.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.317+0000 7fcc1cd9b700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc180fe140 0x7fcc180fc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:48.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.317+0000 7fcc1cd9b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc180fc650 con 0x7fcc180fb0b0 2024-01-23T08:48:48.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.318+0000 7fcc1cd9b700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc180fc7c0 con 0x7fcc180fe140 2024-01-23T08:48:48.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.318+0000 7fcc1659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc180fb0b0 0x7fcc180fdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:48.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.318+0000 7fcc1659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc180fb0b0 0x7fcc180fdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58502/0 (socket says 172.21.15.59:58502) 2024-01-23T08:48:48.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.318+0000 7fcc1659c700 1 -- 172.21.15.59:0/2002954654 learned_addr learned my addr 172.21.15.59:0/2002954654 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:48:48.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.318+0000 7fcc15d9b700 1 --2- 172.21.15.59:0/2002954654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc180fe140 0x7fcc180fc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:48.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.318+0000 7fcc1659c700 1 -- 172.21.15.59:0/2002954654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc180fe140 msgr2=0x7fcc180fc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:48.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.318+0000 7fcc1659c700 1 --2- 172.21.15.59:0/2002954654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc180fe140 0x7fcc180fc050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:48.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.318+0000 7fcc1659c700 1 -- 172.21.15.59:0/2002954654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0c005040 con 0x7fcc180fb0b0 2024-01-23T08:48:48.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.318+0000 7fcc15d9b700 1 --2- 172.21.15.59:0/2002954654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc180fe140 0x7fcc180fc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-23T08:48:48.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.318+0000 7fcc1659c700 1 --2- 172.21.15.59:0/2002954654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc180fb0b0 0x7fcc180fdbd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800c8b0 tx=0x7fcc08004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:48:48.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.318+0000 7fcc06ffd700 1 -- 172.21.15.59:0/2002954654 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc0800a760 con 0x7fcc180fb0b0 2024-01-23T08:48:48.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.318+0000 7fcc06ffd700 1 -- 172.21.15.59:0/2002954654 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc0800ad70 con 0x7fcc180fb0b0 2024-01-23T08:48:48.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.318+0000 7fcc1cd9b700 1 -- 172.21.15.59:0/2002954654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc180fcaa0 con 0x7fcc180fb0b0 2024-01-23T08:48:48.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.318+0000 7fcc06ffd700 1 -- 172.21.15.59:0/2002954654 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc080113a0 con 0x7fcc180fb0b0 2024-01-23T08:48:48.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.318+0000 7fcc1cd9b700 1 -- 172.21.15.59:0/2002954654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc181158c0 con 0x7fcc180fb0b0 2024-01-23T08:48:48.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.319+0000 7fcc1cd9b700 1 -- 172.21.15.59:0/2002954654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc18107700 con 0x7fcc180fb0b0 2024-01-23T08:48:48.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.320+0000 7fcc06ffd700 1 -- 172.21.15.59:0/2002954654 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fcc08012400 con 0x7fcc180fb0b0 2024-01-23T08:48:48.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.320+0000 7fcc06ffd700 1 --2- 172.21.15.59:0/2002954654 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fcc00068f60 0x7fcc0006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:48.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.320+0000 7fcc06ffd700 1 -- 172.21.15.59:0/2002954654 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fcc08086ae0 con 0x7fcc180fb0b0 2024-01-23T08:48:48.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.320+0000 7fcc15d9b700 1 --2- 172.21.15.59:0/2002954654 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fcc00068f60 0x7fcc0006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:48.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.321+0000 7fcc15d9b700 1 --2- 172.21.15.59:0/2002954654 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fcc00068f60 0x7fcc0006b410 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c004f90 tx=0x7fcc0c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:48:48.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.328+0000 7fcc06ffd700 1 -- 172.21.15.59:0/2002954654 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fcc08056d80 con 0x7fcc180fb0b0 2024-01-23T08:48:48.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.598+0000 7fcc1cd9b700 1 -- 172.21.15.59:0/2002954654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fcc1804bd60 con 0x7fcc180fb0b0 2024-01-23T08:48:48.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.602+0000 7fcc06ffd700 1 -- 172.21.15.59:0/2002954654 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7fcc08056910 con 0x7fcc180fb0b0 2024-01-23T08:48:48.599 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-01-23T08:48:48.599 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBwfa9lGe6+IxAAg7G1PjYM5z6R3KlITkenVg== 2024-01-23T08:48:48.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.604+0000 7fcc1cd9b700 1 -- 172.21.15.59:0/2002954654 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fcc00068f60 msgr2=0x7fcc0006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:48.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.604+0000 7fcc1cd9b700 1 --2- 172.21.15.59:0/2002954654 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fcc00068f60 0x7fcc0006b410 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c004f90 tx=0x7fcc0c009e30 comp rx=0 tx=0).stop 2024-01-23T08:48:48.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.604+0000 7fcc1cd9b700 1 -- 172.21.15.59:0/2002954654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc180fb0b0 msgr2=0x7fcc180fdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:48.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.604+0000 7fcc1cd9b700 1 --2- 172.21.15.59:0/2002954654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc180fb0b0 0x7fcc180fdbd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800c8b0 tx=0x7fcc08004470 comp rx=0 tx=0).stop 2024-01-23T08:48:48.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.604+0000 7fcc1cd9b700 1 -- 172.21.15.59:0/2002954654 shutdown_connections 2024-01-23T08:48:48.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.604+0000 7fcc1cd9b700 1 --2- 172.21.15.59:0/2002954654 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fcc00068f60 0x7fcc0006b410 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:48.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.604+0000 7fcc1cd9b700 1 --2- 172.21.15.59:0/2002954654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc180fb0b0 0x7fcc180fdbd0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:48.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.604+0000 7fcc1cd9b700 1 --2- 172.21.15.59:0/2002954654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc180fe140 0x7fcc180fc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:48.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.604+0000 7fcc1cd9b700 1 -- 172.21.15.59:0/2002954654 >> 172.21.15.59:0/2002954654 conn(0x7fcc180f6b30 msgr2=0x7fcc18101c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:48:48.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.605+0000 7fcc1cd9b700 1 -- 172.21.15.59:0/2002954654 shutdown_connections 2024-01-23T08:48:48.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:48.605+0000 7fcc1cd9b700 1 -- 172.21.15.59:0/2002954654 wait complete. 2024-01-23T08:48:49.240 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-23T08:48:49.240 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-01-23T08:48:49.240 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-23T08:48:49.284 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-01-23T08:48:49.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:49 smithi059 ceph-mon[28424]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:48:49.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:49 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/2002954654' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-23T08:48:49.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:49 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/2002954654' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-01-23T08:48:49.548 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi137/config 2024-01-23T08:48:49.670 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:49 smithi137 ceph-mon[35787]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:48:49.670 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:49 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/2002954654' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-23T08:48:49.670 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:49 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/2002954654' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-01-23T08:48:51.181 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.182+0000 7f0d77d0c700 1 -- 172.21.15.137:0/389428652 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d70071d20 msgr2=0x7f0d70070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:51.181 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.182+0000 7f0d77d0c700 1 --2- 172.21.15.137:0/389428652 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d70071d20 0x7f0d70070190 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c009ce0 tx=0x7f0d6c004d10 comp rx=0 tx=0).stop 2024-01-23T08:48:51.181 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.182+0000 7f0d77d0c700 1 -- 172.21.15.137:0/389428652 shutdown_connections 2024-01-23T08:48:51.181 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.182+0000 7f0d77d0c700 1 --2- 172.21.15.137:0/389428652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d700706d0 0x7f0d70070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:51.181 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.182+0000 7f0d77d0c700 1 --2- 172.21.15.137:0/389428652 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d70071d20 0x7f0d70070190 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:51.181 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.182+0000 7f0d77d0c700 1 -- 172.21.15.137:0/389428652 >> 172.21.15.137:0/389428652 conn(0x7f0d700f84b0 msgr2=0x7f0d700fa8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:48:51.181 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.182+0000 7f0d77d0c700 1 -- 172.21.15.137:0/389428652 shutdown_connections 2024-01-23T08:48:51.181 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.182+0000 7f0d77d0c700 1 -- 172.21.15.137:0/389428652 wait complete. 2024-01-23T08:48:51.181 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.183+0000 7f0d77d0c700 1 Processor -- start 2024-01-23T08:48:51.182 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.183+0000 7f0d77d0c700 1 -- start start 2024-01-23T08:48:51.182 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.183+0000 7f0d77d0c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d700706d0 0x7f0d7010faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:51.182 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.183+0000 7f0d77d0c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d70071d20 0x7f0d7010ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:51.182 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.183+0000 7f0d77d0c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d701105e0 con 0x7f0d700706d0 2024-01-23T08:48:51.182 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.183+0000 7f0d77d0c700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d70110e80 con 0x7f0d70071d20 2024-01-23T08:48:51.182 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.183+0000 7f0d752a7700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d70071d20 0x7f0d7010ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:51.182 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.183+0000 7f0d752a7700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d70071d20 0x7f0d7010ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.137:39656/0 (socket says 172.21.15.137:39656) 2024-01-23T08:48:51.182 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.183+0000 7f0d752a7700 1 -- 172.21.15.137:0/619357615 learned_addr learned my addr 172.21.15.137:0/619357615 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-23T08:48:51.182 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.183+0000 7f0d75aa8700 1 --2- 172.21.15.137:0/619357615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d700706d0 0x7f0d7010faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:51.182 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.183+0000 7f0d752a7700 1 -- 172.21.15.137:0/619357615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d700706d0 msgr2=0x7f0d7010faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:51.182 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.183+0000 7f0d752a7700 1 --2- 172.21.15.137:0/619357615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d700706d0 0x7f0d7010faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:51.182 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.183+0000 7f0d752a7700 1 -- 172.21.15.137:0/619357615 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6c009940 con 0x7f0d70071d20 2024-01-23T08:48:51.182 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.183+0000 7f0d752a7700 1 --2- 172.21.15.137:0/619357615 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d70071d20 0x7f0d7010ffe0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0d60002a50 tx=0x7f0d600046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:48:51.182 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.184+0000 7f0d667fc700 1 -- 172.21.15.137:0/619357615 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d60007cc0 con 0x7f0d70071d20 2024-01-23T08:48:51.183 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.184+0000 7f0d77d0c700 1 -- 172.21.15.137:0/619357615 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d70111160 con 0x7f0d70071d20 2024-01-23T08:48:51.183 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.184+0000 7f0d667fc700 1 -- 172.21.15.137:0/619357615 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d6000f040 con 0x7f0d70071d20 2024-01-23T08:48:51.183 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.184+0000 7f0d667fc700 1 -- 172.21.15.137:0/619357615 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d6000d9c0 con 0x7f0d70071d20 2024-01-23T08:48:51.183 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.184+0000 7f0d77d0c700 1 -- 172.21.15.137:0/619357615 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d701116a0 con 0x7f0d70071d20 2024-01-23T08:48:51.184 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.185+0000 7f0d77d0c700 1 -- 172.21.15.137:0/619357615 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d7004bd60 con 0x7f0d70071d20 2024-01-23T08:48:51.184 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.185+0000 7f0d667fc700 1 -- 172.21.15.137:0/619357615 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f0d6000db20 con 0x7f0d70071d20 2024-01-23T08:48:51.184 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.185+0000 7f0d667fc700 1 --2- 172.21.15.137:0/619357615 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f0d5c068f70 0x7f0d5c06b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:51.184 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.186+0000 7f0d667fc700 1 -- 172.21.15.137:0/619357615 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0d60086790 con 0x7f0d70071d20 2024-01-23T08:48:51.184 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.186+0000 7f0d75aa8700 1 --2- 172.21.15.137:0/619357615 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f0d5c068f70 0x7f0d5c06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:51.184 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.186+0000 7f0d75aa8700 1 --2- 172.21.15.137:0/619357615 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f0d5c068f70 0x7f0d5c06b420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c005010 tx=0x7f0d6c004d10 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:48:51.186 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.190+0000 7f0d667fc700 1 -- 172.21.15.137:0/619357615 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0d600569b0 con 0x7f0d70071d20 2024-01-23T08:48:51.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:51 smithi137 ceph-mon[35787]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:48:51.438 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.442+0000 7f0d77d0c700 1 -- 172.21.15.137:0/619357615 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f0d70107950 con 0x7f0d70071d20 2024-01-23T08:48:51.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.447+0000 7f0d667fc700 1 -- 172.21.15.137:0/619357615 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f0d60056540 con 0x7f0d70071d20 2024-01-23T08:48:51.444 INFO:teuthology.orchestra.run.smithi137.stdout:[client.1] 2024-01-23T08:48:51.444 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQBzfa9lxF9wGhAAsbiw6CA7IoL8C/SXcntO2w== 2024-01-23T08:48:51.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.449+0000 7f0d77d0c700 1 -- 172.21.15.137:0/619357615 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f0d5c068f70 msgr2=0x7f0d5c06b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:51.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.449+0000 7f0d77d0c700 1 --2- 172.21.15.137:0/619357615 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f0d5c068f70 0x7f0d5c06b420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c005010 tx=0x7f0d6c004d10 comp rx=0 tx=0).stop 2024-01-23T08:48:51.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.449+0000 7f0d77d0c700 1 -- 172.21.15.137:0/619357615 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d70071d20 msgr2=0x7f0d7010ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:51.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.449+0000 7f0d77d0c700 1 --2- 172.21.15.137:0/619357615 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d70071d20 0x7f0d7010ffe0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0d60002a50 tx=0x7f0d600046b0 comp rx=0 tx=0).stop 2024-01-23T08:48:51.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.449+0000 7f0d77d0c700 1 -- 172.21.15.137:0/619357615 shutdown_connections 2024-01-23T08:48:51.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.449+0000 7f0d77d0c700 1 --2- 172.21.15.137:0/619357615 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f0d5c068f70 0x7f0d5c06b420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:51.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.449+0000 7f0d77d0c700 1 --2- 172.21.15.137:0/619357615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d700706d0 0x7f0d7010faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:51.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.449+0000 7f0d77d0c700 1 --2- 172.21.15.137:0/619357615 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d70071d20 0x7f0d7010ffe0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:51.447 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.449+0000 7f0d77d0c700 1 -- 172.21.15.137:0/619357615 >> 172.21.15.137:0/619357615 conn(0x7f0d700f84b0 msgr2=0x7f0d70102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:48:51.447 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.449+0000 7f0d77d0c700 1 -- 172.21.15.137:0/619357615 shutdown_connections 2024-01-23T08:48:51.447 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-23T08:48:51.449+0000 7f0d77d0c700 1 -- 172.21.15.137:0/619357615 wait complete. 2024-01-23T08:48:51.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:51 smithi059 ceph-mon[28424]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:48:52.010 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-23T08:48:52.010 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-01-23T08:48:52.011 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-01-23T08:48:52.054 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-23T08:48:52.054 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-01-23T08:48:52.054 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph mgr dump --format=json 2024-01-23T08:48:52.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:52 smithi137 ceph-mon[35787]: from='client.? 172.21.15.137:0/619357615' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-23T08:48:52.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:52 smithi137 ceph-mon[35787]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-23T08:48:52.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:52 smithi137 ceph-mon[35787]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-01-23T08:48:52.325 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:48:52.447 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:52 smithi059 ceph-mon[28424]: from='client.? 172.21.15.137:0/619357615' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-23T08:48:52.447 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:52 smithi059 ceph-mon[28424]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-23T08:48:52.447 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:52 smithi059 ceph-mon[28424]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-01-23T08:48:53.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:53 smithi137 ceph-mon[35787]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:48:53.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:53 smithi059 ceph-mon[28424]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:48:54.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.036+0000 7f2421d63700 1 -- 172.21.15.59:0/416643208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c102de0 msgr2=0x7f241c1031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:54.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.036+0000 7f2421d63700 1 --2- 172.21.15.59:0/416643208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c102de0 0x7f241c1031b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f240c004940 tx=0x7f240c009e30 comp rx=0 tx=0).stop 2024-01-23T08:48:54.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.036+0000 7f2421d63700 1 -- 172.21.15.59:0/416643208 shutdown_connections 2024-01-23T08:48:54.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.036+0000 7f2421d63700 1 --2- 172.21.15.59:0/416643208 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f241c103780 0x7f241c0fcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:54.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.036+0000 7f2421d63700 1 --2- 172.21.15.59:0/416643208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c102de0 0x7f241c1031b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:54.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.036+0000 7f2421d63700 1 -- 172.21.15.59:0/416643208 >> 172.21.15.59:0/416643208 conn(0x7f241c0f8970 msgr2=0x7f241c0fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:48:54.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.036+0000 7f2421d63700 1 -- 172.21.15.59:0/416643208 shutdown_connections 2024-01-23T08:48:54.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.036+0000 7f2421d63700 1 -- 172.21.15.59:0/416643208 wait complete. 2024-01-23T08:48:54.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.037+0000 7f2421d63700 1 Processor -- start 2024-01-23T08:48:54.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.037+0000 7f2421d63700 1 -- start start 2024-01-23T08:48:54.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.037+0000 7f2421d63700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c102de0 0x7f241c1138d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:54.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.037+0000 7f2421d63700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f241c103780 0x7f241c113e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:54.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.037+0000 7f2421d63700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c1144f0 con 0x7f241c102de0 2024-01-23T08:48:54.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.037+0000 7f2421d63700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c10d790 con 0x7f241c103780 2024-01-23T08:48:54.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.037+0000 7f241b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c102de0 0x7f241c1138d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:54.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.037+0000 7f241b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c102de0 0x7f241c1138d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58518/0 (socket says 172.21.15.59:58518) 2024-01-23T08:48:54.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.037+0000 7f241b7fe700 1 -- 172.21.15.59:0/1381230634 learned_addr learned my addr 172.21.15.59:0/1381230634 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:48:54.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.037+0000 7f241affd700 1 --2- 172.21.15.59:0/1381230634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f241c103780 0x7f241c113e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:54.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.038+0000 7f241b7fe700 1 -- 172.21.15.59:0/1381230634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f241c103780 msgr2=0x7f241c113e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:54.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.038+0000 7f241b7fe700 1 --2- 172.21.15.59:0/1381230634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f241c103780 0x7f241c113e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:54.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.038+0000 7f241b7fe700 1 -- 172.21.15.59:0/1381230634 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240c005040 con 0x7f241c102de0 2024-01-23T08:48:54.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.038+0000 7f241b7fe700 1 --2- 172.21.15.59:0/1381230634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c102de0 0x7f241c1138d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f240c00b3b0 tx=0x7f240c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:48:54.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.038+0000 7f2403fff700 1 -- 172.21.15.59:0/1381230634 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f240c00e040 con 0x7f241c102de0 2024-01-23T08:48:54.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.038+0000 7f2403fff700 1 -- 172.21.15.59:0/1381230634 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f240c01c430 con 0x7f241c102de0 2024-01-23T08:48:54.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.038+0000 7f2421d63700 1 -- 172.21.15.59:0/1381230634 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f241c10da10 con 0x7f241c102de0 2024-01-23T08:48:54.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.038+0000 7f2403fff700 1 -- 172.21.15.59:0/1381230634 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f240c013640 con 0x7f241c102de0 2024-01-23T08:48:54.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.038+0000 7f2421d63700 1 -- 172.21.15.59:0/1381230634 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f241c10df50 con 0x7f241c102de0 2024-01-23T08:48:54.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.039+0000 7f2421d63700 1 -- 172.21.15.59:0/1381230634 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f241c1000f0 con 0x7f241c102de0 2024-01-23T08:48:54.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.039+0000 7f2403fff700 1 -- 172.21.15.59:0/1381230634 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f240c01c5a0 con 0x7f241c102de0 2024-01-23T08:48:54.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.040+0000 7f2403fff700 1 --2- 172.21.15.59:0/1381230634 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f2404068f60 0x7f240406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:54.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.040+0000 7f2403fff700 1 -- 172.21.15.59:0/1381230634 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f240c087440 con 0x7f241c102de0 2024-01-23T08:48:54.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.040+0000 7f241affd700 1 --2- 172.21.15.59:0/1381230634 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f2404068f60 0x7f240406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:54.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.040+0000 7f241affd700 1 --2- 172.21.15.59:0/1381230634 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f2404068f60 0x7f240406b410 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f241c10f100 tx=0x7f24100091a0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:48:54.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.044+0000 7f2403fff700 1 -- 172.21.15.59:0/1381230634 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f240c057830 con 0x7f241c102de0 2024-01-23T08:48:54.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.294+0000 7f2421d63700 1 -- 172.21.15.59:0/1381230634 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f241c04bd60 con 0x7f241c102de0 2024-01-23T08:48:54.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.298+0000 7f2403fff700 1 -- 172.21.15.59:0/1381230634 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+166776 (secure 0 0 0) 0x7f240c0573c0 con 0x7f241c102de0 2024-01-23T08:48:54.298 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:48:54.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.300+0000 7f2421d63700 1 -- 172.21.15.59:0/1381230634 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f2404068f60 msgr2=0x7f240406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:54.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.300+0000 7f2421d63700 1 --2- 172.21.15.59:0/1381230634 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f2404068f60 0x7f240406b410 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f241c10f100 tx=0x7f24100091a0 comp rx=0 tx=0).stop 2024-01-23T08:48:54.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.300+0000 7f2421d63700 1 -- 172.21.15.59:0/1381230634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c102de0 msgr2=0x7f241c1138d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:54.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.300+0000 7f2421d63700 1 --2- 172.21.15.59:0/1381230634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c102de0 0x7f241c1138d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f240c00b3b0 tx=0x7f240c009e30 comp rx=0 tx=0).stop 2024-01-23T08:48:54.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.300+0000 7f2421d63700 1 -- 172.21.15.59:0/1381230634 shutdown_connections 2024-01-23T08:48:54.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.300+0000 7f2421d63700 1 --2- 172.21.15.59:0/1381230634 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f2404068f60 0x7f240406b410 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:54.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.300+0000 7f2421d63700 1 --2- 172.21.15.59:0/1381230634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c102de0 0x7f241c1138d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:54.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.300+0000 7f2421d63700 1 --2- 172.21.15.59:0/1381230634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f241c103780 0x7f241c113e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:54.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.300+0000 7f2421d63700 1 -- 172.21.15.59:0/1381230634 >> 172.21.15.59:0/1381230634 conn(0x7f241c0f8970 msgr2=0x7f241c0fa530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:48:54.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.300+0000 7f2421d63700 1 -- 172.21.15.59:0/1381230634 shutdown_connections 2024-01-23T08:48:54.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:54.300+0000 7f2421d63700 1 -- 172.21.15.59:0/1381230634 wait complete. 2024-01-23T08:48:54.654 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:54 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/1381230634' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-01-23T08:48:54.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:54 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/1381230634' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-01-23T08:48:55.015 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":19,"active_gid":14210,"active_name":"smithi059.bzqirf","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3219735162},{"type":"v1","addr":"172.21.15.59:6801","nonce":3219735162}]},"active_addr":"172.21.15.59:6801/3219735162","active_change":"2024-01-23T08:45:10.301065+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14232,"name":"smithi137.ooirrh","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.59:8443/","prometheus":"http://172.21.15.59:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.59:0","nonce":283629343}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.59:0","nonce":3983391299}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.59:0","nonce":4236381447}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.59:0","nonce":3725242604}]}]} 2024-01-23T08:48:55.018 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-01-23T08:48:55.018 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-01-23T08:48:55.019 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph osd dump --format=json 2024-01-23T08:48:55.281 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:48:55.717 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:55 smithi059 ceph-mon[28424]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:48:55.717 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:55 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:48:55.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:55 smithi137 ceph-mon[35787]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:48:55.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:55 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:48:56.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.782+0000 7fda53daa700 1 -- 172.21.15.59:0/2107218328 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda4c103780 msgr2=0x7fda4c0fcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:56.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.782+0000 7fda53daa700 1 --2- 172.21.15.59:0/2107218328 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda4c103780 0x7fda4c0fcdf0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fda3c004940 tx=0x7fda3c009e30 comp rx=0 tx=0).stop 2024-01-23T08:48:56.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.782+0000 7fda53daa700 1 -- 172.21.15.59:0/2107218328 shutdown_connections 2024-01-23T08:48:56.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.782+0000 7fda53daa700 1 --2- 172.21.15.59:0/2107218328 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda4c103780 0x7fda4c0fcdf0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:56.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.782+0000 7fda53daa700 1 --2- 172.21.15.59:0/2107218328 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda4c102de0 0x7fda4c1031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:56.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.782+0000 7fda53daa700 1 -- 172.21.15.59:0/2107218328 >> 172.21.15.59:0/2107218328 conn(0x7fda4c0f8970 msgr2=0x7fda4c0fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:48:56.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.783+0000 7fda53daa700 1 -- 172.21.15.59:0/2107218328 shutdown_connections 2024-01-23T08:48:56.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.783+0000 7fda53daa700 1 -- 172.21.15.59:0/2107218328 wait complete. 2024-01-23T08:48:56.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.783+0000 7fda53daa700 1 Processor -- start 2024-01-23T08:48:56.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.783+0000 7fda53daa700 1 -- start start 2024-01-23T08:48:56.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.783+0000 7fda53daa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda4c102de0 0x7fda4c113800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:56.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.783+0000 7fda53daa700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda4c103780 0x7fda4c113d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:56.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.783+0000 7fda53daa700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c114390 con 0x7fda4c102de0 2024-01-23T08:48:56.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.783+0000 7fda53daa700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c10d6c0 con 0x7fda4c103780 2024-01-23T08:48:56.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.783+0000 7fda51b46700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda4c102de0 0x7fda4c113800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:56.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.784+0000 7fda51b46700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda4c102de0 0x7fda4c113800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58542/0 (socket says 172.21.15.59:58542) 2024-01-23T08:48:56.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.784+0000 7fda51b46700 1 -- 172.21.15.59:0/2951677529 learned_addr learned my addr 172.21.15.59:0/2951677529 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:48:56.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.784+0000 7fda51b46700 1 -- 172.21.15.59:0/2951677529 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda4c103780 msgr2=0x7fda4c113d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:56.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.784+0000 7fda51b46700 1 --2- 172.21.15.59:0/2951677529 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda4c103780 0x7fda4c113d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:56.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.784+0000 7fda51b46700 1 -- 172.21.15.59:0/2951677529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c005040 con 0x7fda4c102de0 2024-01-23T08:48:56.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.784+0000 7fda51b46700 1 --2- 172.21.15.59:0/2951677529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda4c102de0 0x7fda4c113800 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fda48002a50 tx=0x7fda480046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:48:56.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.784+0000 7fda427fc700 1 -- 172.21.15.59:0/2951677529 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda48007cc0 con 0x7fda4c102de0 2024-01-23T08:48:56.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.784+0000 7fda53daa700 1 -- 172.21.15.59:0/2951677529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda4c10d9a0 con 0x7fda4c102de0 2024-01-23T08:48:56.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.784+0000 7fda427fc700 1 -- 172.21.15.59:0/2951677529 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda4800f040 con 0x7fda4c102de0 2024-01-23T08:48:56.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.784+0000 7fda427fc700 1 -- 172.21.15.59:0/2951677529 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda48009540 con 0x7fda4c102de0 2024-01-23T08:48:56.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.784+0000 7fda53daa700 1 -- 172.21.15.59:0/2951677529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda4c10dee0 con 0x7fda4c102de0 2024-01-23T08:48:56.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.785+0000 7fda53daa700 1 -- 172.21.15.59:0/2951677529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda4c1052b0 con 0x7fda4c102de0 2024-01-23T08:48:56.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.785+0000 7fda427fc700 1 -- 172.21.15.59:0/2951677529 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fda480096a0 con 0x7fda4c102de0 2024-01-23T08:48:56.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.786+0000 7fda427fc700 1 --2- 172.21.15.59:0/2951677529 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fda38068f60 0x7fda3806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:56.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.786+0000 7fda427fc700 1 -- 172.21.15.59:0/2951677529 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fda48086860 con 0x7fda4c102de0 2024-01-23T08:48:56.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.786+0000 7fda51345700 1 --2- 172.21.15.59:0/2951677529 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fda38068f60 0x7fda3806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:56.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.786+0000 7fda51345700 1 --2- 172.21.15.59:0/2951677529 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fda38068f60 0x7fda3806b410 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fda3c00b3b0 tx=0x7fda3c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:48:56.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.790+0000 7fda427fc700 1 -- 172.21.15.59:0/2951677529 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fda48056a80 con 0x7fda4c102de0 2024-01-23T08:48:56.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.980+0000 7fda53daa700 1 -- 172.21.15.59:0/2951677529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda4c04bd60 con 0x7fda4c102de0 2024-01-23T08:48:56.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.981+0000 7fda427fc700 1 -- 172.21.15.59:0/2951677529 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11069 (secure 0 0 0) 0x7fda48056610 con 0x7fda4c102de0 2024-01-23T08:48:56.979 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:48:56.979 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":41,"fsid":"3fc07a38-b9cb-11ee-95b1-87774f69a715","created":"2024-01-23T08:43:05.316045+0000","modified":"2024-01-23T08:48:37.419823+0000","last_up_change":"2024-01-23T08:48:36.414344+0000","last_in_change":"2024-01-23T08:48:17.747223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-23T08:47:24.641141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"444a9c2f-e580-443e-9dfe-36843f8c2e13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2564662034},{"type":"v1","addr":"172.21.15.59:6803","nonce":2564662034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2564662034},{"type":"v1","addr":"172.21.15.59:6805","nonce":2564662034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2564662034},{"type":"v1","addr":"172.21.15.59:6809","nonce":2564662034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2564662034},{"type":"v1","addr":"172.21.15.59:6807","nonce":2564662034}]},"public_addr":"172.21.15.59:6803/2564662034","cluster_addr":"172.21.15.59:6805/2564662034","heartbeat_back_addr":"172.21.15.59:6809/2564662034","heartbeat_front_addr":"172.21.15.59:6807/2564662034","state":["exists","up"]},{"osd":1,"uuid":"c28ec1b9-2f5f-48e6-8322-7092bbe8cdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1673825133},{"type":"v1","addr":"172.21.15.59:6811","nonce":1673825133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1673825133},{"type":"v1","addr":"172.21.15.59:6813","nonce":1673825133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1673825133},{"type":"v1","addr":"172.21.15.59:6817","nonce":1673825133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1673825133},{"type":"v1","addr":"172.21.15.59:6815","nonce":1673825133}]},"public_addr":"172.21.15.59:6811/1673825133","cluster_addr":"172.21.15.59:6813/1673825133","heartbeat_back_addr":"172.21.15.59:6817/1673825133","heartbeat_front_addr":"172.21.15.59:6815/1673825133","state":["exists","up"]},{"osd":2,"uuid":"94d9e85d-6f22-4e83-9f4c-b326e9375e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2143964017},{"type":"v1","addr":"172.21.15.59:6819","nonce":2143964017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2143964017},{"type":"v1","addr":"172.21.15.59:6821","nonce":2143964017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2143964017},{"type":"v1","addr":"172.21.15.59:6825","nonce":2143964017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2143964017},{"type":"v1","addr":"172.21.15.59:6823","nonce":2143964017}]},"public_addr":"172.21.15.59:6819/2143964017","cluster_addr":"172.21.15.59:6821/2143964017","heartbeat_back_addr":"172.21.15.59:6825/2143964017","heartbeat_front_addr":"172.21.15.59:6823/2143964017","state":["exists","up"]},{"osd":3,"uuid":"19dfe941-4ce2-4490-8a09-bf85493593bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2062812031},{"type":"v1","addr":"172.21.15.137:6801","nonce":2062812031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2062812031},{"type":"v1","addr":"172.21.15.137:6803","nonce":2062812031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2062812031},{"type":"v1","addr":"172.21.15.137:6807","nonce":2062812031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2062812031},{"type":"v1","addr":"172.21.15.137:6805","nonce":2062812031}]},"public_addr":"172.21.15.137:6801/2062812031","cluster_addr":"172.21.15.137:6803/2062812031","heartbeat_back_addr":"172.21.15.137:6807/2062812031","heartbeat_front_addr":"172.21.15.137:6805/2062812031","state":["exists","up"]},{"osd":4,"uuid":"a1d6209f-095c-42e2-966a-11559d090531","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":708078156},{"type":"v1","addr":"172.21.15.137:6809","nonce":708078156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":708078156},{"type":"v1","addr":"172.21.15.137:6811","nonce":708078156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":708078156},{"type":"v1","addr":"172.21.15.137:6815","nonce":708078156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":708078156},{"type":"v1","addr":"172.21.15.137:6813","nonce":708078156}]},"public_addr":"172.21.15.137:6809/708078156","cluster_addr":"172.21.15.137:6811/708078156","heartbeat_back_addr":"172.21.15.137:6815/708078156","heartbeat_front_addr":"172.21.15.137:6813/708078156","state":["exists","up"]},{"osd":5,"uuid":"fdaab2d1-80f0-4928-817e-8ac45394367e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":765377139},{"type":"v1","addr":"172.21.15.137:6817","nonce":765377139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":765377139},{"type":"v1","addr":"172.21.15.137:6819","nonce":765377139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":765377139},{"type":"v1","addr":"172.21.15.137:6823","nonce":765377139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":765377139},{"type":"v1","addr":"172.21.15.137:6821","nonce":765377139}]},"public_addr":"172.21.15.137:6817/765377139","cluster_addr":"172.21.15.137:6819/765377139","heartbeat_back_addr":"172.21.15.137:6823/765377139","heartbeat_front_addr":"172.21.15.137:6821/765377139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-23T08:46:25.064905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-23T08:46:52.940650+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-23T08:47:21.360904+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-23T08:47:43.875916+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-23T08:48:09.457117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-23T08:48:34.887400+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:0/2423485094":"2024-01-24T08:45:10.300954+0000","172.21.15.59:0/1598234521":"2024-01-24T08:45:10.300954+0000","172.21.15.59:6801/3614124404":"2024-01-24T08:45:10.300954+0000","172.21.15.59:6800/3614124404":"2024-01-24T08:45:10.300954+0000","172.21.15.59:0/2596349924":"2024-01-24T08:44:03.688872+0000","172.21.15.59:0/3153116091":"2024-01-24T08:44:03.688872+0000","172.21.15.59:6801/1337467262":"2024-01-24T08:44:03.688872+0000","172.21.15.59:6800/1337467262":"2024-01-24T08:44:03.688872+0000","172.21.15.59:0/1368475691":"2024-01-24T08:43:31.660943+0000","172.21.15.59:0/1345586333":"2024-01-24T08:43:31.660943+0000","172.21.15.59:6801/504555577":"2024-01-24T08:43:31.660943+0000","172.21.15.59:0/252476549":"2024-01-24T08:43:31.660943+0000","172.21.15.59:0/2405882784":"2024-01-24T08:45:10.300954+0000","172.21.15.59:6800/504555577":"2024-01-24T08:43:31.660943+0000","172.21.15.59:0/715244053":"2024-01-24T08:44:03.688872+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-23T08:48:56.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.983+0000 7fda53daa700 1 -- 172.21.15.59:0/2951677529 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fda38068f60 msgr2=0x7fda3806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:56.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.983+0000 7fda53daa700 1 --2- 172.21.15.59:0/2951677529 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fda38068f60 0x7fda3806b410 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fda3c00b3b0 tx=0x7fda3c009e30 comp rx=0 tx=0).stop 2024-01-23T08:48:56.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.983+0000 7fda53daa700 1 -- 172.21.15.59:0/2951677529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda4c102de0 msgr2=0x7fda4c113800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:56.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.983+0000 7fda53daa700 1 --2- 172.21.15.59:0/2951677529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda4c102de0 0x7fda4c113800 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fda48002a50 tx=0x7fda480046b0 comp rx=0 tx=0).stop 2024-01-23T08:48:56.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.983+0000 7fda53daa700 1 -- 172.21.15.59:0/2951677529 shutdown_connections 2024-01-23T08:48:56.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.983+0000 7fda53daa700 1 --2- 172.21.15.59:0/2951677529 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fda38068f60 0x7fda3806b410 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:56.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.983+0000 7fda53daa700 1 --2- 172.21.15.59:0/2951677529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda4c102de0 0x7fda4c113800 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:56.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.983+0000 7fda53daa700 1 --2- 172.21.15.59:0/2951677529 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda4c103780 0x7fda4c113d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:56.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.983+0000 7fda53daa700 1 -- 172.21.15.59:0/2951677529 >> 172.21.15.59:0/2951677529 conn(0x7fda4c0f8970 msgr2=0x7fda4c0fa510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:48:56.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.983+0000 7fda53daa700 1 -- 172.21.15.59:0/2951677529 shutdown_connections 2024-01-23T08:48:56.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:56.983+0000 7fda53daa700 1 -- 172.21.15.59:0/2951677529 wait complete. 2024-01-23T08:48:57.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:57 smithi059 ceph-mon[28424]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:48:57.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:57 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/2951677529' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-23T08:48:57.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:57 smithi137 ceph-mon[35787]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:48:57.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:57 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/2951677529' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-23T08:48:57.897 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-01-23T08:48:57.897 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph osd dump --format=json 2024-01-23T08:48:58.158 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:48:59.530 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:48:59 smithi059 ceph-mon[28424]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:48:59.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.747+0000 7fe921da0700 1 -- 172.21.15.59:0/1985128261 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe91c101ce0 msgr2=0x7fe91c1020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:59.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.747+0000 7fe921da0700 1 --2- 172.21.15.59:0/1985128261 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe91c101ce0 0x7fe91c1020b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe90c004940 tx=0x7fe90c009e30 comp rx=0 tx=0).stop 2024-01-23T08:48:59.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.747+0000 7fe921da0700 1 -- 172.21.15.59:0/1985128261 shutdown_connections 2024-01-23T08:48:59.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.747+0000 7fe921da0700 1 --2- 172.21.15.59:0/1985128261 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe91c102680 0x7fe91c0fbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:59.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.747+0000 7fe921da0700 1 --2- 172.21.15.59:0/1985128261 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe91c101ce0 0x7fe91c1020b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:59.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.747+0000 7fe921da0700 1 -- 172.21.15.59:0/1985128261 >> 172.21.15.59:0/1985128261 conn(0x7fe91c0f7850 msgr2=0x7fe91c0f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:48:59.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.748+0000 7fe921da0700 1 -- 172.21.15.59:0/1985128261 shutdown_connections 2024-01-23T08:48:59.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.748+0000 7fe921da0700 1 -- 172.21.15.59:0/1985128261 wait complete. 2024-01-23T08:48:59.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.748+0000 7fe921da0700 1 Processor -- start 2024-01-23T08:48:59.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.748+0000 7fe921da0700 1 -- start start 2024-01-23T08:48:59.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.748+0000 7fe921da0700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe91c101ce0 0x7fe91c113820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:59.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.749+0000 7fe921da0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe91c102680 0x7fe91c113d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:59.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.749+0000 7fe91b7fe700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe91c101ce0 0x7fe91c113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:59.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.749+0000 7fe921da0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91c1143b0 con 0x7fe91c102680 2024-01-23T08:48:59.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.749+0000 7fe921da0700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91c10d6e0 con 0x7fe91c101ce0 2024-01-23T08:48:59.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.749+0000 7fe91affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe91c102680 0x7fe91c113d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:59.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.749+0000 7fe91b7fe700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe91c101ce0 0x7fe91c113820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.59:52796/0 (socket says 172.21.15.59:52796) 2024-01-23T08:48:59.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.749+0000 7fe91affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe91c102680 0x7fe91c113d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33564/0 (socket says 172.21.15.59:33564) 2024-01-23T08:48:59.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.749+0000 7fe91b7fe700 1 -- 172.21.15.59:0/1541332948 learned_addr learned my addr 172.21.15.59:0/1541332948 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:48:59.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.749+0000 7fe91affd700 1 -- 172.21.15.59:0/1541332948 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe91c101ce0 msgr2=0x7fe91c113820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:59.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.749+0000 7fe91affd700 1 --2- 172.21.15.59:0/1541332948 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe91c101ce0 0x7fe91c113820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:59.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.749+0000 7fe91affd700 1 -- 172.21.15.59:0/1541332948 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe90c005040 con 0x7fe91c102680 2024-01-23T08:48:59.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.749+0000 7fe91b7fe700 1 --2- 172.21.15.59:0/1541332948 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe91c101ce0 0x7fe91c113820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-23T08:48:59.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.750+0000 7fe91affd700 1 --2- 172.21.15.59:0/1541332948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe91c102680 0x7fe91c113d60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe91c067260 tx=0x7fe910004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:48:59.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.750+0000 7fe903fff700 1 -- 172.21.15.59:0/1541332948 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9100106f0 con 0x7fe91c102680 2024-01-23T08:48:59.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.750+0000 7fe921da0700 1 -- 172.21.15.59:0/1541332948 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe91c10d9c0 con 0x7fe91c102680 2024-01-23T08:48:59.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.750+0000 7fe903fff700 1 -- 172.21.15.59:0/1541332948 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe910010d00 con 0x7fe91c102680 2024-01-23T08:48:59.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.750+0000 7fe903fff700 1 -- 172.21.15.59:0/1541332948 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe910012ea0 con 0x7fe91c102680 2024-01-23T08:48:59.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.750+0000 7fe921da0700 1 -- 172.21.15.59:0/1541332948 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe91c10df00 con 0x7fe91c102680 2024-01-23T08:48:59.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.751+0000 7fe921da0700 1 -- 172.21.15.59:0/1541332948 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe91c0feff0 con 0x7fe91c102680 2024-01-23T08:48:59.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.751+0000 7fe903fff700 1 -- 172.21.15.59:0/1541332948 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fe910010850 con 0x7fe91c102680 2024-01-23T08:48:59.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.752+0000 7fe903fff700 1 --2- 172.21.15.59:0/1541332948 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fe904068f70 0x7fe90406b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:48:59.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.752+0000 7fe903fff700 1 -- 172.21.15.59:0/1541332948 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe910086a50 con 0x7fe91c102680 2024-01-23T08:48:59.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.752+0000 7fe91b7fe700 1 --2- 172.21.15.59:0/1541332948 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fe904068f70 0x7fe90406b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:48:59.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.752+0000 7fe91b7fe700 1 --2- 172.21.15.59:0/1541332948 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fe904068f70 0x7fe90406b420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe90c000c00 tx=0x7fe90c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:48:59.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.756+0000 7fe903fff700 1 -- 172.21.15.59:0/1541332948 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe910056c50 con 0x7fe91c102680 2024-01-23T08:48:59.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:48:59 smithi137 ceph-mon[35787]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:48:59.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.947+0000 7fe921da0700 1 -- 172.21.15.59:0/1541332948 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe91c0fb470 con 0x7fe91c102680 2024-01-23T08:48:59.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.948+0000 7fe903fff700 1 -- 172.21.15.59:0/1541332948 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11069 (secure 0 0 0) 0x7fe9100567e0 con 0x7fe91c102680 2024-01-23T08:48:59.945 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:48:59.946 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":41,"fsid":"3fc07a38-b9cb-11ee-95b1-87774f69a715","created":"2024-01-23T08:43:05.316045+0000","modified":"2024-01-23T08:48:37.419823+0000","last_up_change":"2024-01-23T08:48:36.414344+0000","last_in_change":"2024-01-23T08:48:17.747223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-23T08:47:24.641141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"444a9c2f-e580-443e-9dfe-36843f8c2e13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2564662034},{"type":"v1","addr":"172.21.15.59:6803","nonce":2564662034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2564662034},{"type":"v1","addr":"172.21.15.59:6805","nonce":2564662034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2564662034},{"type":"v1","addr":"172.21.15.59:6809","nonce":2564662034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2564662034},{"type":"v1","addr":"172.21.15.59:6807","nonce":2564662034}]},"public_addr":"172.21.15.59:6803/2564662034","cluster_addr":"172.21.15.59:6805/2564662034","heartbeat_back_addr":"172.21.15.59:6809/2564662034","heartbeat_front_addr":"172.21.15.59:6807/2564662034","state":["exists","up"]},{"osd":1,"uuid":"c28ec1b9-2f5f-48e6-8322-7092bbe8cdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1673825133},{"type":"v1","addr":"172.21.15.59:6811","nonce":1673825133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1673825133},{"type":"v1","addr":"172.21.15.59:6813","nonce":1673825133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1673825133},{"type":"v1","addr":"172.21.15.59:6817","nonce":1673825133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1673825133},{"type":"v1","addr":"172.21.15.59:6815","nonce":1673825133}]},"public_addr":"172.21.15.59:6811/1673825133","cluster_addr":"172.21.15.59:6813/1673825133","heartbeat_back_addr":"172.21.15.59:6817/1673825133","heartbeat_front_addr":"172.21.15.59:6815/1673825133","state":["exists","up"]},{"osd":2,"uuid":"94d9e85d-6f22-4e83-9f4c-b326e9375e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2143964017},{"type":"v1","addr":"172.21.15.59:6819","nonce":2143964017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2143964017},{"type":"v1","addr":"172.21.15.59:6821","nonce":2143964017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2143964017},{"type":"v1","addr":"172.21.15.59:6825","nonce":2143964017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2143964017},{"type":"v1","addr":"172.21.15.59:6823","nonce":2143964017}]},"public_addr":"172.21.15.59:6819/2143964017","cluster_addr":"172.21.15.59:6821/2143964017","heartbeat_back_addr":"172.21.15.59:6825/2143964017","heartbeat_front_addr":"172.21.15.59:6823/2143964017","state":["exists","up"]},{"osd":3,"uuid":"19dfe941-4ce2-4490-8a09-bf85493593bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2062812031},{"type":"v1","addr":"172.21.15.137:6801","nonce":2062812031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2062812031},{"type":"v1","addr":"172.21.15.137:6803","nonce":2062812031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2062812031},{"type":"v1","addr":"172.21.15.137:6807","nonce":2062812031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2062812031},{"type":"v1","addr":"172.21.15.137:6805","nonce":2062812031}]},"public_addr":"172.21.15.137:6801/2062812031","cluster_addr":"172.21.15.137:6803/2062812031","heartbeat_back_addr":"172.21.15.137:6807/2062812031","heartbeat_front_addr":"172.21.15.137:6805/2062812031","state":["exists","up"]},{"osd":4,"uuid":"a1d6209f-095c-42e2-966a-11559d090531","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":708078156},{"type":"v1","addr":"172.21.15.137:6809","nonce":708078156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":708078156},{"type":"v1","addr":"172.21.15.137:6811","nonce":708078156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":708078156},{"type":"v1","addr":"172.21.15.137:6815","nonce":708078156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":708078156},{"type":"v1","addr":"172.21.15.137:6813","nonce":708078156}]},"public_addr":"172.21.15.137:6809/708078156","cluster_addr":"172.21.15.137:6811/708078156","heartbeat_back_addr":"172.21.15.137:6815/708078156","heartbeat_front_addr":"172.21.15.137:6813/708078156","state":["exists","up"]},{"osd":5,"uuid":"fdaab2d1-80f0-4928-817e-8ac45394367e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":765377139},{"type":"v1","addr":"172.21.15.137:6817","nonce":765377139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":765377139},{"type":"v1","addr":"172.21.15.137:6819","nonce":765377139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":765377139},{"type":"v1","addr":"172.21.15.137:6823","nonce":765377139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":765377139},{"type":"v1","addr":"172.21.15.137:6821","nonce":765377139}]},"public_addr":"172.21.15.137:6817/765377139","cluster_addr":"172.21.15.137:6819/765377139","heartbeat_back_addr":"172.21.15.137:6823/765377139","heartbeat_front_addr":"172.21.15.137:6821/765377139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-23T08:46:25.064905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-23T08:46:52.940650+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-23T08:47:21.360904+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-23T08:47:43.875916+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-23T08:48:09.457117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-23T08:48:34.887400+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:0/2423485094":"2024-01-24T08:45:10.300954+0000","172.21.15.59:0/1598234521":"2024-01-24T08:45:10.300954+0000","172.21.15.59:6801/3614124404":"2024-01-24T08:45:10.300954+0000","172.21.15.59:6800/3614124404":"2024-01-24T08:45:10.300954+0000","172.21.15.59:0/2596349924":"2024-01-24T08:44:03.688872+0000","172.21.15.59:0/3153116091":"2024-01-24T08:44:03.688872+0000","172.21.15.59:6801/1337467262":"2024-01-24T08:44:03.688872+0000","172.21.15.59:6800/1337467262":"2024-01-24T08:44:03.688872+0000","172.21.15.59:0/1368475691":"2024-01-24T08:43:31.660943+0000","172.21.15.59:0/1345586333":"2024-01-24T08:43:31.660943+0000","172.21.15.59:6801/504555577":"2024-01-24T08:43:31.660943+0000","172.21.15.59:0/252476549":"2024-01-24T08:43:31.660943+0000","172.21.15.59:0/2405882784":"2024-01-24T08:45:10.300954+0000","172.21.15.59:6800/504555577":"2024-01-24T08:43:31.660943+0000","172.21.15.59:0/715244053":"2024-01-24T08:44:03.688872+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-23T08:48:59.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.950+0000 7fe921da0700 1 -- 172.21.15.59:0/1541332948 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fe904068f70 msgr2=0x7fe90406b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:59.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.950+0000 7fe921da0700 1 --2- 172.21.15.59:0/1541332948 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fe904068f70 0x7fe90406b420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe90c000c00 tx=0x7fe90c009e30 comp rx=0 tx=0).stop 2024-01-23T08:48:59.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.950+0000 7fe921da0700 1 -- 172.21.15.59:0/1541332948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe91c102680 msgr2=0x7fe91c113d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:48:59.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.950+0000 7fe921da0700 1 --2- 172.21.15.59:0/1541332948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe91c102680 0x7fe91c113d60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe91c067260 tx=0x7fe910004430 comp rx=0 tx=0).stop 2024-01-23T08:48:59.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.950+0000 7fe921da0700 1 -- 172.21.15.59:0/1541332948 shutdown_connections 2024-01-23T08:48:59.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.950+0000 7fe921da0700 1 --2- 172.21.15.59:0/1541332948 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fe904068f70 0x7fe90406b420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:59.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.950+0000 7fe921da0700 1 --2- 172.21.15.59:0/1541332948 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe91c101ce0 0x7fe91c113820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:59.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.950+0000 7fe921da0700 1 --2- 172.21.15.59:0/1541332948 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe91c102680 0x7fe91c113d60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:48:59.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.950+0000 7fe921da0700 1 -- 172.21.15.59:0/1541332948 >> 172.21.15.59:0/1541332948 conn(0x7fe91c0f7850 msgr2=0x7fe91c0f93d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:48:59.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.950+0000 7fe921da0700 1 -- 172.21.15.59:0/1541332948 shutdown_connections 2024-01-23T08:48:59.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:48:59.950+0000 7fe921da0700 1 -- 172.21.15.59:0/1541332948 wait complete. 2024-01-23T08:49:00.530 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:00 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/1541332948' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-23T08:49:00.532 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-01-23T08:49:00.532 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-01-23T08:49:00.533 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-01-23T08:49:00.533 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-01-23T08:49:00.534 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-01-23T08:49:00.534 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-01-23T08:49:00.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:00 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/1541332948' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-23T08:49:00.835 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:49:00.835 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:49:01.529 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:01 smithi059 ceph-mon[28424]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:01.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:01 smithi137 ceph-mon[35787]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:03.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.405+0000 7fe228802700 1 -- 172.21.15.59:0/991637558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe224102c20 msgr2=0x7fe224102ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:03.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.405+0000 7fe228802700 1 --2- 172.21.15.59:0/991637558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe224102c20 0x7fe224102ff0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe214004940 tx=0x7fe214009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:03.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.406+0000 7fe228802700 1 -- 172.21.15.59:0/991637558 shutdown_connections 2024-01-23T08:49:03.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.406+0000 7fe228802700 1 --2- 172.21.15.59:0/991637558 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe2241035c0 0x7fe2240fcc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:03.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.406+0000 7fe228802700 1 --2- 172.21.15.59:0/991637558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe224102c20 0x7fe224102ff0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:03.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.406+0000 7fe228802700 1 -- 172.21.15.59:0/991637558 >> 172.21.15.59:0/991637558 conn(0x7fe2240f87d0 msgr2=0x7fe2240fabe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:03.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.406+0000 7fe228802700 1 -- 172.21.15.59:0/991637558 shutdown_connections 2024-01-23T08:49:03.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.406+0000 7fe228802700 1 -- 172.21.15.59:0/991637558 wait complete. 2024-01-23T08:49:03.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.406+0000 7fe228802700 1 Processor -- start 2024-01-23T08:49:03.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.406+0000 7fe228802700 1 -- start start 2024-01-23T08:49:03.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.407+0000 7fe228802700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe224102c20 0x7fe224070120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:03.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.407+0000 7fe228802700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2241035c0 0x7fe224070660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:03.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.407+0000 7fe228802700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe224070c60 con 0x7fe2241035c0 2024-01-23T08:49:03.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.407+0000 7fe228802700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe224070dd0 con 0x7fe224102c20 2024-01-23T08:49:03.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.407+0000 7fe22259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2241035c0 0x7fe224070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:03.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.407+0000 7fe22259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2241035c0 0x7fe224070660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33576/0 (socket says 172.21.15.59:33576) 2024-01-23T08:49:03.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.407+0000 7fe22259c700 1 -- 172.21.15.59:0/481140920 learned_addr learned my addr 172.21.15.59:0/481140920 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:49:03.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.407+0000 7fe222d9d700 1 --2- 172.21.15.59:0/481140920 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe224102c20 0x7fe224070120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:03.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.407+0000 7fe22259c700 1 -- 172.21.15.59:0/481140920 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe224102c20 msgr2=0x7fe224070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:03.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.407+0000 7fe22259c700 1 --2- 172.21.15.59:0/481140920 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe224102c20 0x7fe224070120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:03.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.407+0000 7fe22259c700 1 -- 172.21.15.59:0/481140920 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe214005040 con 0x7fe2241035c0 2024-01-23T08:49:03.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.407+0000 7fe222d9d700 1 --2- 172.21.15.59:0/481140920 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe224102c20 0x7fe224070120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-23T08:49:03.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.407+0000 7fe22259c700 1 --2- 172.21.15.59:0/481140920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2241035c0 0x7fe224070660 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe20c00c7e0 tx=0x7fe20c007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:03.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.407+0000 7fe21b7fe700 1 -- 172.21.15.59:0/481140920 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe20c012810 con 0x7fe2241035c0 2024-01-23T08:49:03.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.408+0000 7fe228802700 1 -- 172.21.15.59:0/481140920 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe224119850 con 0x7fe2241035c0 2024-01-23T08:49:03.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.408+0000 7fe21b7fe700 1 -- 172.21.15.59:0/481140920 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe20c012e20 con 0x7fe2241035c0 2024-01-23T08:49:03.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.408+0000 7fe21b7fe700 1 -- 172.21.15.59:0/481140920 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe20c00a310 con 0x7fe2241035c0 2024-01-23T08:49:03.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.408+0000 7fe228802700 1 -- 172.21.15.59:0/481140920 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe224119db0 con 0x7fe2241035c0 2024-01-23T08:49:03.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.408+0000 7fe228802700 1 -- 172.21.15.59:0/481140920 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe2240fac40 con 0x7fe2241035c0 2024-01-23T08:49:03.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.409+0000 7fe21b7fe700 1 -- 172.21.15.59:0/481140920 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fe20c011380 con 0x7fe2241035c0 2024-01-23T08:49:03.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.409+0000 7fe21b7fe700 1 --2- 172.21.15.59:0/481140920 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fe210068fc0 0x7fe21006b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:03.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.410+0000 7fe222d9d700 1 --2- 172.21.15.59:0/481140920 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fe210068fc0 0x7fe21006b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:03.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.410+0000 7fe21b7fe700 1 -- 172.21.15.59:0/481140920 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe20c086eb0 con 0x7fe2241035c0 2024-01-23T08:49:03.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.410+0000 7fe21b7fe700 1 --2- 172.21.15.59:0/481140920 >> [v2:172.21.15.59:6810/1673825133,v1:172.21.15.59:6811/1673825133] conn(0x7fe21006ec90 0x7fe2100710a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:03.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.410+0000 7fe21b7fe700 1 -- 172.21.15.59:0/481140920 --> [v2:172.21.15.59:6810/1673825133,v1:172.21.15.59:6811/1673825133] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe210071750 con 0x7fe21006ec90 2024-01-23T08:49:03.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.410+0000 7fe21b7fe700 1 -- 172.21.15.59:0/481140920 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe20c08c4b0 con 0x7fe2241035c0 2024-01-23T08:49:03.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.410+0000 7fe22359e700 1 --2- 172.21.15.59:0/481140920 >> [v2:172.21.15.59:6810/1673825133,v1:172.21.15.59:6811/1673825133] conn(0x7fe21006ec90 0x7fe2100710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:03.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.410+0000 7fe222d9d700 1 --2- 172.21.15.59:0/481140920 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fe210068fc0 0x7fe21006b470 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe21400b3b0 tx=0x7fe214009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:03.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.410+0000 7fe22359e700 1 --2- 172.21.15.59:0/481140920 >> [v2:172.21.15.59:6810/1673825133,v1:172.21.15.59:6811/1673825133] conn(0x7fe21006ec90 0x7fe2100710a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:03.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.411+0000 7fe21b7fe700 1 -- 172.21.15.59:0/481140920 <== osd.1 v2:172.21.15.59:6810/1673825133 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fe210071750 con 0x7fe21006ec90 2024-01-23T08:49:03.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.426+0000 7fe228802700 1 -- 172.21.15.59:0/481140920 --> [v2:172.21.15.59:6810/1673825133,v1:172.21.15.59:6811/1673825133] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe224070f90 con 0x7fe21006ec90 2024-01-23T08:49:03.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.427+0000 7fe21b7fe700 1 -- 172.21.15.59:0/481140920 <== osd.1 v2:172.21.15.59:6810/1673825133 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe224070f90 con 0x7fe21006ec90 2024-01-23T08:49:03.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.427+0000 7fe228802700 1 -- 172.21.15.59:0/481140920 >> [v2:172.21.15.59:6810/1673825133,v1:172.21.15.59:6811/1673825133] conn(0x7fe21006ec90 msgr2=0x7fe2100710a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:03.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.427+0000 7fe228802700 1 --2- 172.21.15.59:0/481140920 >> [v2:172.21.15.59:6810/1673825133,v1:172.21.15.59:6811/1673825133] conn(0x7fe21006ec90 0x7fe2100710a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:03.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.427+0000 7fe228802700 1 -- 172.21.15.59:0/481140920 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fe210068fc0 msgr2=0x7fe21006b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:03.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.427+0000 7fe228802700 1 --2- 172.21.15.59:0/481140920 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fe210068fc0 0x7fe21006b470 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe21400b3b0 tx=0x7fe214009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:03.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.427+0000 7fe228802700 1 -- 172.21.15.59:0/481140920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2241035c0 msgr2=0x7fe224070660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:03.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.427+0000 7fe228802700 1 --2- 172.21.15.59:0/481140920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2241035c0 0x7fe224070660 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe20c00c7e0 tx=0x7fe20c007f90 comp rx=0 tx=0).stop 2024-01-23T08:49:03.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.427+0000 7fe228802700 1 -- 172.21.15.59:0/481140920 shutdown_connections 2024-01-23T08:49:03.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.427+0000 7fe228802700 1 --2- 172.21.15.59:0/481140920 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fe210068fc0 0x7fe21006b470 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:03.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.427+0000 7fe228802700 1 --2- 172.21.15.59:0/481140920 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe224102c20 0x7fe224070120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:03.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.427+0000 7fe228802700 1 --2- 172.21.15.59:0/481140920 >> [v2:172.21.15.59:6810/1673825133,v1:172.21.15.59:6811/1673825133] conn(0x7fe21006ec90 0x7fe2100710a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:03.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.427+0000 7fe228802700 1 --2- 172.21.15.59:0/481140920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2241035c0 0x7fe224070660 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:03.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.427+0000 7fe228802700 1 -- 172.21.15.59:0/481140920 >> 172.21.15.59:0/481140920 conn(0x7fe2240f87d0 msgr2=0x7fe2240fa230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:03.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.427+0000 7fe228802700 1 -- 172.21.15.59:0/481140920 shutdown_connections 2024-01-23T08:49:03.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:03.427+0000 7fe228802700 1 -- 172.21.15.59:0/481140920 wait complete. 2024-01-23T08:49:03.664 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:03 smithi059 ceph-mon[28424]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:03.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:03 smithi137 ceph-mon[35787]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:04.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.054+0000 7fb034f8d700 1 -- 172.21.15.59:0/687787029 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0300fdce0 msgr2=0x7fb0300fe0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:04.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.054+0000 7fb034f8d700 1 --2- 172.21.15.59:0/687787029 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0300fdce0 0x7fb0300fe0b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb020004940 tx=0x7fb020009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:04.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.054+0000 7fb034f8d700 1 -- 172.21.15.59:0/687787029 shutdown_connections 2024-01-23T08:49:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.054+0000 7fb034f8d700 1 --2- 172.21.15.59:0/687787029 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0300fb400 0x7fb0300fb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.054+0000 7fb034f8d700 1 --2- 172.21.15.59:0/687787029 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0300fdce0 0x7fb0300fe0b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.054+0000 7fb034f8d700 1 -- 172.21.15.59:0/687787029 >> 172.21.15.59:0/687787029 conn(0x7fb0300713f0 msgr2=0x7fb0300717f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.054+0000 7fb034f8d700 1 -- 172.21.15.59:0/687787029 shutdown_connections 2024-01-23T08:49:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.054+0000 7fb034f8d700 1 -- 172.21.15.59:0/687787029 wait complete. 2024-01-23T08:49:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.055+0000 7fb034f8d700 1 Processor -- start 2024-01-23T08:49:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.055+0000 7fb034f8d700 1 -- start start 2024-01-23T08:49:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.055+0000 7fb034f8d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0300fb400 0x7fb0301159e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.055+0000 7fb034f8d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0300fdce0 0x7fb030115f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.055+0000 7fb034f8d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb030116570 con 0x7fb0300fdce0 2024-01-23T08:49:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.055+0000 7fb034f8d700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03010f8a0 con 0x7fb0300fb400 2024-01-23T08:49:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.055+0000 7fb02dd9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0300fdce0 0x7fb030115f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.055+0000 7fb02dd9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0300fdce0 0x7fb030115f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33590/0 (socket says 172.21.15.59:33590) 2024-01-23T08:49:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.055+0000 7fb02dd9b700 1 -- 172.21.15.59:0/2872154411 learned_addr learned my addr 172.21.15.59:0/2872154411 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:49:04.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.055+0000 7fb02e59c700 1 --2- 172.21.15.59:0/2872154411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0300fb400 0x7fb0301159e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:04.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.056+0000 7fb02dd9b700 1 -- 172.21.15.59:0/2872154411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0300fb400 msgr2=0x7fb0301159e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:04.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.056+0000 7fb02dd9b700 1 --2- 172.21.15.59:0/2872154411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0300fb400 0x7fb0301159e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:04.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.056+0000 7fb02dd9b700 1 -- 172.21.15.59:0/2872154411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb020005040 con 0x7fb0300fdce0 2024-01-23T08:49:04.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.056+0000 7fb02dd9b700 1 --2- 172.21.15.59:0/2872154411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0300fdce0 0x7fb030115f20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb02400c830 tx=0x7fb024004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:04.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.056+0000 7fb01effd700 1 -- 172.21.15.59:0/2872154411 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0240106f0 con 0x7fb0300fdce0 2024-01-23T08:49:04.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.056+0000 7fb01effd700 1 -- 172.21.15.59:0/2872154411 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb024010d00 con 0x7fb0300fdce0 2024-01-23T08:49:04.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.056+0000 7fb01effd700 1 -- 172.21.15.59:0/2872154411 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb024012e90 con 0x7fb0300fdce0 2024-01-23T08:49:04.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.056+0000 7fb034f8d700 1 -- 172.21.15.59:0/2872154411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb03010fb80 con 0x7fb0300fdce0 2024-01-23T08:49:04.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.056+0000 7fb034f8d700 1 -- 172.21.15.59:0/2872154411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb030110070 con 0x7fb0300fdce0 2024-01-23T08:49:04.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.057+0000 7fb034f8d700 1 -- 172.21.15.59:0/2872154411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb030110640 con 0x7fb0300fdce0 2024-01-23T08:49:04.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.057+0000 7fb01effd700 1 -- 172.21.15.59:0/2872154411 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fb024010850 con 0x7fb0300fdce0 2024-01-23T08:49:04.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.058+0000 7fb01effd700 1 --2- 172.21.15.59:0/2872154411 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fb018068f70 0x7fb01806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:04.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.058+0000 7fb01effd700 1 -- 172.21.15.59:0/2872154411 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb024086a30 con 0x7fb0300fdce0 2024-01-23T08:49:04.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.058+0000 7fb02e59c700 1 --2- 172.21.15.59:0/2872154411 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fb018068f70 0x7fb01806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:04.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.058+0000 7fb01effd700 1 --2- 172.21.15.59:0/2872154411 >> [v2:172.21.15.137:6816/765377139,v1:172.21.15.137:6817/765377139] conn(0x7fb01806ead0 0x7fb018070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:04.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.058+0000 7fb01effd700 1 -- 172.21.15.59:0/2872154411 --> [v2:172.21.15.137:6816/765377139,v1:172.21.15.137:6817/765377139] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb018071590 con 0x7fb01806ead0 2024-01-23T08:49:04.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.058+0000 7fb01effd700 1 -- 172.21.15.59:0/2872154411 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb02408b220 con 0x7fb0300fdce0 2024-01-23T08:49:04.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.058+0000 7fb02e59c700 1 --2- 172.21.15.59:0/2872154411 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fb018068f70 0x7fb01806b420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb02000b3b0 tx=0x7fb020009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:04.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.058+0000 7fb02ed9d700 1 --2- 172.21.15.59:0/2872154411 >> [v2:172.21.15.137:6816/765377139,v1:172.21.15.137:6817/765377139] conn(0x7fb01806ead0 0x7fb018070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:04.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.059+0000 7fb02ed9d700 1 --2- 172.21.15.59:0/2872154411 >> [v2:172.21.15.137:6816/765377139,v1:172.21.15.137:6817/765377139] conn(0x7fb01806ead0 0x7fb018070ee0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:04.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.059+0000 7fb01effd700 1 -- 172.21.15.59:0/2872154411 <== osd.5 v2:172.21.15.137:6816/765377139 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fb018071590 con 0x7fb01806ead0 2024-01-23T08:49:04.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.074+0000 7fb034f8d700 1 -- 172.21.15.59:0/2872154411 --> [v2:172.21.15.137:6816/765377139,v1:172.21.15.137:6817/765377139] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb03005e4a0 con 0x7fb01806ead0 2024-01-23T08:49:04.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.075+0000 7fb01effd700 1 -- 172.21.15.59:0/2872154411 <== osd.5 v2:172.21.15.137:6816/765377139 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb03005e4a0 con 0x7fb01806ead0 2024-01-23T08:49:04.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.075+0000 7fb034f8d700 1 -- 172.21.15.59:0/2872154411 >> [v2:172.21.15.137:6816/765377139,v1:172.21.15.137:6817/765377139] conn(0x7fb01806ead0 msgr2=0x7fb018070ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:04.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.075+0000 7fb034f8d700 1 --2- 172.21.15.59:0/2872154411 >> [v2:172.21.15.137:6816/765377139,v1:172.21.15.137:6817/765377139] conn(0x7fb01806ead0 0x7fb018070ee0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:04.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.075+0000 7fb034f8d700 1 -- 172.21.15.59:0/2872154411 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fb018068f70 msgr2=0x7fb01806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:04.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.075+0000 7fb034f8d700 1 --2- 172.21.15.59:0/2872154411 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fb018068f70 0x7fb01806b420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb02000b3b0 tx=0x7fb020009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:04.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.075+0000 7fb034f8d700 1 -- 172.21.15.59:0/2872154411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0300fdce0 msgr2=0x7fb030115f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:04.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.075+0000 7fb034f8d700 1 --2- 172.21.15.59:0/2872154411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0300fdce0 0x7fb030115f20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb02400c830 tx=0x7fb024004470 comp rx=0 tx=0).stop 2024-01-23T08:49:04.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.075+0000 7fb034f8d700 1 -- 172.21.15.59:0/2872154411 shutdown_connections 2024-01-23T08:49:04.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.075+0000 7fb034f8d700 1 --2- 172.21.15.59:0/2872154411 >> [v2:172.21.15.137:6816/765377139,v1:172.21.15.137:6817/765377139] conn(0x7fb01806ead0 0x7fb018070ee0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:04.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.075+0000 7fb034f8d700 1 --2- 172.21.15.59:0/2872154411 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fb018068f70 0x7fb01806b420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:04.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.075+0000 7fb034f8d700 1 --2- 172.21.15.59:0/2872154411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0300fb400 0x7fb0301159e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:04.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.075+0000 7fb034f8d700 1 --2- 172.21.15.59:0/2872154411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0300fdce0 0x7fb030115f20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:04.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.075+0000 7fb034f8d700 1 -- 172.21.15.59:0/2872154411 >> 172.21.15.59:0/2872154411 conn(0x7fb0300713f0 msgr2=0x7fb0300f9590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:04.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.075+0000 7fb034f8d700 1 -- 172.21.15.59:0/2872154411 shutdown_connections 2024-01-23T08:49:04.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:04.075+0000 7fb034f8d700 1 -- 172.21.15.59:0/2872154411 wait complete. 2024-01-23T08:49:04.625 INFO:teuthology.orchestra.run.smithi059.stdout:171798691849 2024-01-23T08:49:04.625 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-01-23T08:49:04.636 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:49:04.637 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:49:04.637 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:49:04.638 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:49:05.664 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:05 smithi059 ceph-mon[28424]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:05.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:05 smithi137 ceph-mon[35787]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:06.716 INFO:teuthology.orchestra.run.smithi059.stdout:60129542172 2024-01-23T08:49:06.716 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-01-23T08:49:07.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:07 smithi059 ceph-mon[28424]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:07.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:07 smithi137 ceph-mon[35787]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:08.580 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:49:08.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.613+0000 7f193b355700 1 -- 172.21.15.59:0/2275703012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1934102680 msgr2=0x7f19340fbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:08.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.613+0000 7f193b355700 1 --2- 172.21.15.59:0/2275703012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1934102680 0x7f19340fbcf0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1924004940 tx=0x7f1924009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:08.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.613+0000 7f193b355700 1 -- 172.21.15.59:0/2275703012 shutdown_connections 2024-01-23T08:49:08.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.613+0000 7f193b355700 1 --2- 172.21.15.59:0/2275703012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1934102680 0x7f19340fbcf0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:08.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.613+0000 7f193b355700 1 --2- 172.21.15.59:0/2275703012 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1934101ce0 0x7f19341020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:08.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.613+0000 7f193b355700 1 -- 172.21.15.59:0/2275703012 >> 172.21.15.59:0/2275703012 conn(0x7f19340f7870 msgr2=0x7f19340f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:08.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.613+0000 7f193b355700 1 -- 172.21.15.59:0/2275703012 shutdown_connections 2024-01-23T08:49:08.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.613+0000 7f193b355700 1 -- 172.21.15.59:0/2275703012 wait complete. 2024-01-23T08:49:08.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.614+0000 7f193b355700 1 Processor -- start 2024-01-23T08:49:08.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.614+0000 7f193b355700 1 -- start start 2024-01-23T08:49:08.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.614+0000 7f193b355700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1934101ce0 0x7f19341124b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:08.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.614+0000 7f193b355700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1934102680 0x7f19341129f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:08.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.614+0000 7f193b355700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1934113040 con 0x7f1934101ce0 2024-01-23T08:49:08.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.614+0000 7f193b355700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193410c580 con 0x7f1934102680 2024-01-23T08:49:08.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.614+0000 7f19390f1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1934101ce0 0x7f19341124b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:08.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.614+0000 7f19390f1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1934101ce0 0x7f19341124b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36800/0 (socket says 172.21.15.59:36800) 2024-01-23T08:49:08.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.614+0000 7f19390f1700 1 -- 172.21.15.59:0/99674047 learned_addr learned my addr 172.21.15.59:0/99674047 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:49:08.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.614+0000 7f19388f0700 1 --2- 172.21.15.59:0/99674047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1934102680 0x7f19341129f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:08.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.615+0000 7f19390f1700 1 -- 172.21.15.59:0/99674047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1934102680 msgr2=0x7f19341129f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:08.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.615+0000 7f19390f1700 1 --2- 172.21.15.59:0/99674047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1934102680 0x7f19341129f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:08.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.615+0000 7f19390f1700 1 -- 172.21.15.59:0/99674047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1924005040 con 0x7f1934101ce0 2024-01-23T08:49:08.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.615+0000 7f19390f1700 1 --2- 172.21.15.59:0/99674047 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1934101ce0 0x7f19341124b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1930002a50 tx=0x7f19300046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:08.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.615+0000 7f1929ffb700 1 -- 172.21.15.59:0/99674047 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1930007cc0 con 0x7f1934101ce0 2024-01-23T08:49:08.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.615+0000 7f193b355700 1 -- 172.21.15.59:0/99674047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f193410c860 con 0x7f1934101ce0 2024-01-23T08:49:08.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.615+0000 7f1929ffb700 1 -- 172.21.15.59:0/99674047 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f193000f040 con 0x7f1934101ce0 2024-01-23T08:49:08.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.615+0000 7f1929ffb700 1 -- 172.21.15.59:0/99674047 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1930009540 con 0x7f1934101ce0 2024-01-23T08:49:08.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.615+0000 7f193b355700 1 -- 172.21.15.59:0/99674047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f193410cd70 con 0x7f1934101ce0 2024-01-23T08:49:08.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.616+0000 7f193b355700 1 -- 172.21.15.59:0/99674047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f193410d250 con 0x7f1934101ce0 2024-01-23T08:49:08.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.616+0000 7f1929ffb700 1 -- 172.21.15.59:0/99674047 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f19300096a0 con 0x7f1934101ce0 2024-01-23T08:49:08.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.617+0000 7f1929ffb700 1 --2- 172.21.15.59:0/99674047 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f1920068f70 0x7f192006b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:08.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.617+0000 7f1929ffb700 1 -- 172.21.15.59:0/99674047 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1930086890 con 0x7f1934101ce0 2024-01-23T08:49:08.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.617+0000 7f19388f0700 1 --2- 172.21.15.59:0/99674047 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f1920068f70 0x7f192006b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:08.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.617+0000 7f1929ffb700 1 --2- 172.21.15.59:0/99674047 >> [v2:172.21.15.59:6818/2143964017,v1:172.21.15.59:6819/2143964017] conn(0x7f192006ec40 0x7f1920071050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:08.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.617+0000 7f1929ffb700 1 -- 172.21.15.59:0/99674047 --> [v2:172.21.15.59:6818/2143964017,v1:172.21.15.59:6819/2143964017] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1920071700 con 0x7f192006ec40 2024-01-23T08:49:08.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.617+0000 7f1929ffb700 1 -- 172.21.15.59:0/99674047 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f193008b220 con 0x7f1934101ce0 2024-01-23T08:49:08.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.617+0000 7f19398f2700 1 --2- 172.21.15.59:0/99674047 >> [v2:172.21.15.59:6818/2143964017,v1:172.21.15.59:6819/2143964017] conn(0x7f192006ec40 0x7f1920071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:08.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.617+0000 7f19388f0700 1 --2- 172.21.15.59:0/99674047 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f1920068f70 0x7f192006b420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f193410d8b0 tx=0x7f1924009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:08.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.618+0000 7f19398f2700 1 --2- 172.21.15.59:0/99674047 >> [v2:172.21.15.59:6818/2143964017,v1:172.21.15.59:6819/2143964017] conn(0x7f192006ec40 0x7f1920071050 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:08.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.618+0000 7f1929ffb700 1 -- 172.21.15.59:0/99674047 <== osd.2 v2:172.21.15.59:6818/2143964017 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f1920071700 con 0x7f192006ec40 2024-01-23T08:49:08.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.633+0000 7f193b355700 1 -- 172.21.15.59:0/99674047 --> [v2:172.21.15.59:6818/2143964017,v1:172.21.15.59:6819/2143964017] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f193405e520 con 0x7f192006ec40 2024-01-23T08:49:08.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.633+0000 7f1929ffb700 1 -- 172.21.15.59:0/99674047 <== osd.2 v2:172.21.15.59:6818/2143964017 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f193405e520 con 0x7f192006ec40 2024-01-23T08:49:08.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.633+0000 7f193b355700 1 -- 172.21.15.59:0/99674047 >> [v2:172.21.15.59:6818/2143964017,v1:172.21.15.59:6819/2143964017] conn(0x7f192006ec40 msgr2=0x7f1920071050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:08.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.633+0000 7f193b355700 1 --2- 172.21.15.59:0/99674047 >> [v2:172.21.15.59:6818/2143964017,v1:172.21.15.59:6819/2143964017] conn(0x7f192006ec40 0x7f1920071050 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:08.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.634+0000 7f193b355700 1 -- 172.21.15.59:0/99674047 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f1920068f70 msgr2=0x7f192006b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:08.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.634+0000 7f193b355700 1 --2- 172.21.15.59:0/99674047 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f1920068f70 0x7f192006b420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f193410d8b0 tx=0x7f1924009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:08.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.634+0000 7f193b355700 1 -- 172.21.15.59:0/99674047 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1934101ce0 msgr2=0x7f19341124b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:08.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.634+0000 7f193b355700 1 --2- 172.21.15.59:0/99674047 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1934101ce0 0x7f19341124b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1930002a50 tx=0x7f19300046b0 comp rx=0 tx=0).stop 2024-01-23T08:49:08.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.634+0000 7f193b355700 1 -- 172.21.15.59:0/99674047 shutdown_connections 2024-01-23T08:49:08.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.634+0000 7f193b355700 1 --2- 172.21.15.59:0/99674047 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f1920068f70 0x7f192006b420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:08.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.634+0000 7f193b355700 1 --2- 172.21.15.59:0/99674047 >> [v2:172.21.15.59:6818/2143964017,v1:172.21.15.59:6819/2143964017] conn(0x7f192006ec40 0x7f1920071050 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:08.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.634+0000 7f193b355700 1 --2- 172.21.15.59:0/99674047 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1934101ce0 0x7f19341124b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:08.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.634+0000 7f193b355700 1 --2- 172.21.15.59:0/99674047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1934102680 0x7f19341129f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:08.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.634+0000 7f193b355700 1 -- 172.21.15.59:0/99674047 >> 172.21.15.59:0/99674047 conn(0x7f19340f7870 msgr2=0x7f19340f9360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:08.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.634+0000 7f193b355700 1 -- 172.21.15.59:0/99674047 shutdown_connections 2024-01-23T08:49:08.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:08.634+0000 7f193b355700 1 -- 172.21.15.59:0/99674047 wait complete. 2024-01-23T08:49:09.508 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:09 smithi059 ceph-mon[28424]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:09.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:09 smithi137 ceph-mon[35787]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:11.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.260+0000 7f1283a14700 1 -- 172.21.15.59:0/1425174338 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f127c101cd0 msgr2=0x7f127c1020a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:11.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.260+0000 7f1283a14700 1 --2- 172.21.15.59:0/1425174338 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f127c101cd0 0x7f127c1020a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f126c009ce0 tx=0x7f126c004d10 comp rx=0 tx=0).stop 2024-01-23T08:49:11.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.260+0000 7f1283a14700 1 -- 172.21.15.59:0/1425174338 shutdown_connections 2024-01-23T08:49:11.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.260+0000 7f1283a14700 1 --2- 172.21.15.59:0/1425174338 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f127c102670 0x7f127c0fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:11.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.260+0000 7f1283a14700 1 --2- 172.21.15.59:0/1425174338 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f127c101cd0 0x7f127c1020a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:11.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.260+0000 7f1283a14700 1 -- 172.21.15.59:0/1425174338 >> 172.21.15.59:0/1425174338 conn(0x7f127c071c20 msgr2=0x7f127c072020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:11.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.260+0000 7f1283a14700 1 -- 172.21.15.59:0/1425174338 shutdown_connections 2024-01-23T08:49:11.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.260+0000 7f1283a14700 1 -- 172.21.15.59:0/1425174338 wait complete. 2024-01-23T08:49:11.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.260+0000 7f1283a14700 1 Processor -- start 2024-01-23T08:49:11.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.260+0000 7f1283a14700 1 -- start start 2024-01-23T08:49:11.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.261+0000 7f1283a14700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f127c101cd0 0x7f127c10ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:11.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.261+0000 7f1283a14700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f127c102670 0x7f127c10ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:11.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.261+0000 7f1283a14700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127c10f5b0 con 0x7f127c102670 2024-01-23T08:49:11.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.261+0000 7f1283a14700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127c10f720 con 0x7f127c101cd0 2024-01-23T08:49:11.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.261+0000 7f1280faf700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f127c102670 0x7f127c10ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:11.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.261+0000 7f1280faf700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f127c102670 0x7f127c10ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36832/0 (socket says 172.21.15.59:36832) 2024-01-23T08:49:11.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.261+0000 7f1280faf700 1 -- 172.21.15.59:0/724502023 learned_addr learned my addr 172.21.15.59:0/724502023 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:49:11.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.261+0000 7f12817b0700 1 --2- 172.21.15.59:0/724502023 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f127c101cd0 0x7f127c10ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:11.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.261+0000 7f1280faf700 1 -- 172.21.15.59:0/724502023 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f127c101cd0 msgr2=0x7f127c10ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:11.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.261+0000 7f1280faf700 1 --2- 172.21.15.59:0/724502023 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f127c101cd0 0x7f127c10ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:11.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.261+0000 7f1280faf700 1 -- 172.21.15.59:0/724502023 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126c009940 con 0x7f127c102670 2024-01-23T08:49:11.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.261+0000 7f1280faf700 1 --2- 172.21.15.59:0/724502023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f127c102670 0x7f127c10ef70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f127000b790 tx=0x7f1270009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:11.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.261+0000 7f1279ffb700 1 -- 172.21.15.59:0/724502023 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f127000f6f0 con 0x7f127c102670 2024-01-23T08:49:11.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.261+0000 7f1279ffb700 1 -- 172.21.15.59:0/724502023 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f127000fd00 con 0x7f127c102670 2024-01-23T08:49:11.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.261+0000 7f1283a14700 1 -- 172.21.15.59:0/724502023 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f127c117a80 con 0x7f127c102670 2024-01-23T08:49:11.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.261+0000 7f1279ffb700 1 -- 172.21.15.59:0/724502023 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12700103a0 con 0x7f127c102670 2024-01-23T08:49:11.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.262+0000 7f1283a14700 1 -- 172.21.15.59:0/724502023 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f127c117fc0 con 0x7f127c102670 2024-01-23T08:49:11.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.262+0000 7f1283a14700 1 -- 172.21.15.59:0/724502023 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f127c05e4e0 con 0x7f127c102670 2024-01-23T08:49:11.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.263+0000 7f1279ffb700 1 -- 172.21.15.59:0/724502023 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f127000f850 con 0x7f127c102670 2024-01-23T08:49:11.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.263+0000 7f1279ffb700 1 --2- 172.21.15.59:0/724502023 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f1264068f60 0x7f126406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:11.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.263+0000 7f1279ffb700 1 -- 172.21.15.59:0/724502023 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f12700867d0 con 0x7f127c102670 2024-01-23T08:49:11.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.263+0000 7f12817b0700 1 --2- 172.21.15.59:0/724502023 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f1264068f60 0x7f126406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:11.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.263+0000 7f1279ffb700 1 --2- 172.21.15.59:0/724502023 >> [v2:172.21.15.137:6808/708078156,v1:172.21.15.137:6809/708078156] conn(0x7f126406ec30 0x7f1264071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:11.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.263+0000 7f1279ffb700 1 -- 172.21.15.59:0/724502023 --> [v2:172.21.15.137:6808/708078156,v1:172.21.15.137:6809/708078156] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f12640716f0 con 0x7f126406ec30 2024-01-23T08:49:11.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.263+0000 7f1279ffb700 1 -- 172.21.15.59:0/724502023 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1270088d60 con 0x7f127c102670 2024-01-23T08:49:11.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.263+0000 7f12817b0700 1 --2- 172.21.15.59:0/724502023 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f1264068f60 0x7f126406b410 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f126c015040 tx=0x7f126c004d10 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:11.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.264+0000 7f1281fb1700 1 --2- 172.21.15.59:0/724502023 >> [v2:172.21.15.137:6808/708078156,v1:172.21.15.137:6809/708078156] conn(0x7f126406ec30 0x7f1264071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:11.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.264+0000 7f1281fb1700 1 --2- 172.21.15.59:0/724502023 >> [v2:172.21.15.137:6808/708078156,v1:172.21.15.137:6809/708078156] conn(0x7f126406ec30 0x7f1264071040 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:11.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.265+0000 7f1279ffb700 1 -- 172.21.15.59:0/724502023 <== osd.4 v2:172.21.15.137:6808/708078156 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f12640716f0 con 0x7f126406ec30 2024-01-23T08:49:11.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.280+0000 7f1283a14700 1 -- 172.21.15.59:0/724502023 --> [v2:172.21.15.137:6808/708078156,v1:172.21.15.137:6809/708078156] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f127c05e910 con 0x7f126406ec30 2024-01-23T08:49:11.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.280+0000 7f1279ffb700 1 -- 172.21.15.59:0/724502023 <== osd.4 v2:172.21.15.137:6808/708078156 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f127c05e910 con 0x7f126406ec30 2024-01-23T08:49:11.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.281+0000 7f1283a14700 1 -- 172.21.15.59:0/724502023 >> [v2:172.21.15.137:6808/708078156,v1:172.21.15.137:6809/708078156] conn(0x7f126406ec30 msgr2=0x7f1264071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:11.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.281+0000 7f1283a14700 1 --2- 172.21.15.59:0/724502023 >> [v2:172.21.15.137:6808/708078156,v1:172.21.15.137:6809/708078156] conn(0x7f126406ec30 0x7f1264071040 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:11.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.281+0000 7f1283a14700 1 -- 172.21.15.59:0/724502023 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f1264068f60 msgr2=0x7f126406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:11.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.281+0000 7f1283a14700 1 --2- 172.21.15.59:0/724502023 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f1264068f60 0x7f126406b410 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f126c015040 tx=0x7f126c004d10 comp rx=0 tx=0).stop 2024-01-23T08:49:11.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.281+0000 7f1283a14700 1 -- 172.21.15.59:0/724502023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f127c102670 msgr2=0x7f127c10ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:11.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.281+0000 7f1283a14700 1 --2- 172.21.15.59:0/724502023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f127c102670 0x7f127c10ef70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f127000b790 tx=0x7f1270009220 comp rx=0 tx=0).stop 2024-01-23T08:49:11.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.281+0000 7f1283a14700 1 -- 172.21.15.59:0/724502023 shutdown_connections 2024-01-23T08:49:11.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.281+0000 7f1283a14700 1 --2- 172.21.15.59:0/724502023 >> [v2:172.21.15.137:6808/708078156,v1:172.21.15.137:6809/708078156] conn(0x7f126406ec30 0x7f1264071040 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:11.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.281+0000 7f1283a14700 1 --2- 172.21.15.59:0/724502023 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f1264068f60 0x7f126406b410 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:11.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.281+0000 7f1283a14700 1 --2- 172.21.15.59:0/724502023 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f127c101cd0 0x7f127c10ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:11.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.281+0000 7f1283a14700 1 --2- 172.21.15.59:0/724502023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f127c102670 0x7f127c10ef70 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:11.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.281+0000 7f1283a14700 1 -- 172.21.15.59:0/724502023 >> 172.21.15.59:0/724502023 conn(0x7f127c071c20 msgr2=0x7f127c0fac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:11.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.281+0000 7f1283a14700 1 -- 172.21.15.59:0/724502023 shutdown_connections 2024-01-23T08:49:11.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.281+0000 7f1283a14700 1 -- 172.21.15.59:0/724502023 wait complete. 2024-01-23T08:49:11.288 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:11 smithi059 ceph-mon[28424]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:11.288 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:11 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:49:11.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:11 smithi137 ceph-mon[35787]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:11.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:11 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:49:11.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.996+0000 7f4d211f1700 1 -- 172.21.15.59:0/1973552609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d1c0fe780 msgr2=0x7f4d1c106ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:11.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.996+0000 7f4d211f1700 1 --2- 172.21.15.59:0/1973552609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d1c0fe780 0x7f4d1c106ee0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4d10004970 tx=0x7f4d10009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:11.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.996+0000 7f4d211f1700 1 -- 172.21.15.59:0/1973552609 shutdown_connections 2024-01-23T08:49:11.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.996+0000 7f4d211f1700 1 --2- 172.21.15.59:0/1973552609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d1c0fe780 0x7f4d1c106ee0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:11.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.996+0000 7f4d211f1700 1 --2- 172.21.15.59:0/1973552609 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d1c0fde70 0x7f4d1c0fe240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:11.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.996+0000 7f4d211f1700 1 -- 172.21.15.59:0/1973552609 >> 172.21.15.59:0/1973552609 conn(0x7f4d1c071c20 msgr2=0x7f4d1c072020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:11.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.996+0000 7f4d211f1700 1 -- 172.21.15.59:0/1973552609 shutdown_connections 2024-01-23T08:49:11.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.996+0000 7f4d211f1700 1 -- 172.21.15.59:0/1973552609 wait complete. 2024-01-23T08:49:11.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.997+0000 7f4d211f1700 1 Processor -- start 2024-01-23T08:49:11.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.997+0000 7f4d211f1700 1 -- start start 2024-01-23T08:49:11.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.997+0000 7f4d211f1700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d1c0fde70 0x7f4d1c0fbd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:11.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.997+0000 7f4d211f1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d1c0fe780 0x7f4d1c0fc270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:11.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.997+0000 7f4d211f1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c0fc870 con 0x7f4d1c0fe780 2024-01-23T08:49:11.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.997+0000 7f4d211f1700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c0fc9e0 con 0x7f4d1c0fde70 2024-01-23T08:49:11.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.997+0000 7f4d1a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d1c0fe780 0x7f4d1c0fc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:11.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.997+0000 7f4d1a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d1c0fe780 0x7f4d1c0fc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36862/0 (socket says 172.21.15.59:36862) 2024-01-23T08:49:11.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.997+0000 7f4d1a59c700 1 -- 172.21.15.59:0/3705434652 learned_addr learned my addr 172.21.15.59:0/3705434652 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:49:11.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.997+0000 7f4d1ad9d700 1 --2- 172.21.15.59:0/3705434652 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d1c0fde70 0x7f4d1c0fbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:11.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.997+0000 7f4d1a59c700 1 -- 172.21.15.59:0/3705434652 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d1c0fde70 msgr2=0x7f4d1c0fbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:11.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.997+0000 7f4d1a59c700 1 --2- 172.21.15.59:0/3705434652 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d1c0fde70 0x7f4d1c0fbd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:11.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.997+0000 7f4d1a59c700 1 -- 172.21.15.59:0/3705434652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d10005040 con 0x7f4d1c0fe780 2024-01-23T08:49:11.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.998+0000 7f4d1a59c700 1 --2- 172.21.15.59:0/3705434652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d1c0fe780 0x7f4d1c0fc270 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4d10000c00 tx=0x7f4d10009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:11.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.998+0000 7f4d037fe700 1 -- 172.21.15.59:0/3705434652 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d1000e040 con 0x7f4d1c0fe780 2024-01-23T08:49:11.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.998+0000 7f4d211f1700 1 -- 172.21.15.59:0/3705434652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d1c119a40 con 0x7f4d1c0fe780 2024-01-23T08:49:11.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.998+0000 7f4d037fe700 1 -- 172.21.15.59:0/3705434652 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d10013420 con 0x7f4d1c0fe780 2024-01-23T08:49:11.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.998+0000 7f4d037fe700 1 -- 172.21.15.59:0/3705434652 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d1001c630 con 0x7f4d1c0fe780 2024-01-23T08:49:11.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.998+0000 7f4d211f1700 1 -- 172.21.15.59:0/3705434652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d1c119ea0 con 0x7f4d1c0fe780 2024-01-23T08:49:11.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.999+0000 7f4d211f1700 1 -- 172.21.15.59:0/3705434652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4d1c05e4e0 con 0x7f4d1c0fe780 2024-01-23T08:49:11.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:11.999+0000 7f4d037fe700 1 -- 172.21.15.59:0/3705434652 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f4d100139e0 con 0x7f4d1c0fe780 2024-01-23T08:49:11.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.000+0000 7f4d037fe700 1 --2- 172.21.15.59:0/3705434652 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f4d04068f60 0x7f4d0406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:11.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.000+0000 7f4d037fe700 1 -- 172.21.15.59:0/3705434652 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4d100887e0 con 0x7f4d1c0fe780 2024-01-23T08:49:11.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.000+0000 7f4d1ad9d700 1 --2- 172.21.15.59:0/3705434652 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f4d04068f60 0x7f4d0406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:11.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.000+0000 7f4d037fe700 1 --2- 172.21.15.59:0/3705434652 >> [v2:172.21.15.137:6800/2062812031,v1:172.21.15.137:6801/2062812031] conn(0x7f4d0406ec30 0x7f4d04071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:11.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.000+0000 7f4d037fe700 1 -- 172.21.15.59:0/3705434652 --> [v2:172.21.15.137:6800/2062812031,v1:172.21.15.137:6801/2062812031] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d040716f0 con 0x7f4d0406ec30 2024-01-23T08:49:11.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.000+0000 7f4d037fe700 1 -- 172.21.15.59:0/3705434652 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d10088bc0 con 0x7f4d1c0fe780 2024-01-23T08:49:11.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.000+0000 7f4d1ad9d700 1 --2- 172.21.15.59:0/3705434652 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f4d04068f60 0x7f4d0406b410 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c004f40 tx=0x7f4d0c00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:11.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.000+0000 7f4d1b59e700 1 --2- 172.21.15.59:0/3705434652 >> [v2:172.21.15.137:6800/2062812031,v1:172.21.15.137:6801/2062812031] conn(0x7f4d0406ec30 0x7f4d04071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:11.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.001+0000 7f4d1b59e700 1 --2- 172.21.15.59:0/3705434652 >> [v2:172.21.15.137:6800/2062812031,v1:172.21.15.137:6801/2062812031] conn(0x7f4d0406ec30 0x7f4d04071040 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:11.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.002+0000 7f4d037fe700 1 -- 172.21.15.59:0/3705434652 <== osd.3 v2:172.21.15.137:6800/2062812031 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f4d040716f0 con 0x7f4d0406ec30 2024-01-23T08:49:12.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.016+0000 7f4d211f1700 1 -- 172.21.15.59:0/3705434652 --> [v2:172.21.15.137:6800/2062812031,v1:172.21.15.137:6801/2062812031] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4d1c05e8e0 con 0x7f4d0406ec30 2024-01-23T08:49:12.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.017+0000 7f4d037fe700 1 -- 172.21.15.59:0/3705434652 <== osd.3 v2:172.21.15.137:6800/2062812031 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4d1c05e8e0 con 0x7f4d0406ec30 2024-01-23T08:49:12.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.017+0000 7f4d211f1700 1 -- 172.21.15.59:0/3705434652 >> [v2:172.21.15.137:6800/2062812031,v1:172.21.15.137:6801/2062812031] conn(0x7f4d0406ec30 msgr2=0x7f4d04071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:12.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.017+0000 7f4d211f1700 1 --2- 172.21.15.59:0/3705434652 >> [v2:172.21.15.137:6800/2062812031,v1:172.21.15.137:6801/2062812031] conn(0x7f4d0406ec30 0x7f4d04071040 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:12.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.017+0000 7f4d211f1700 1 -- 172.21.15.59:0/3705434652 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f4d04068f60 msgr2=0x7f4d0406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:12.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.017+0000 7f4d211f1700 1 --2- 172.21.15.59:0/3705434652 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f4d04068f60 0x7f4d0406b410 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c004f40 tx=0x7f4d0c00b040 comp rx=0 tx=0).stop 2024-01-23T08:49:12.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.017+0000 7f4d211f1700 1 -- 172.21.15.59:0/3705434652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d1c0fe780 msgr2=0x7f4d1c0fc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:12.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.017+0000 7f4d211f1700 1 --2- 172.21.15.59:0/3705434652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d1c0fe780 0x7f4d1c0fc270 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4d10000c00 tx=0x7f4d10009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:12.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.018+0000 7f4d211f1700 1 -- 172.21.15.59:0/3705434652 shutdown_connections 2024-01-23T08:49:12.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.018+0000 7f4d211f1700 1 --2- 172.21.15.59:0/3705434652 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f4d04068f60 0x7f4d0406b410 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:12.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.018+0000 7f4d211f1700 1 --2- 172.21.15.59:0/3705434652 >> [v2:172.21.15.137:6800/2062812031,v1:172.21.15.137:6801/2062812031] conn(0x7f4d0406ec30 0x7f4d04071040 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:12.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.018+0000 7f4d211f1700 1 --2- 172.21.15.59:0/3705434652 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d1c0fde70 0x7f4d1c0fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:12.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.018+0000 7f4d211f1700 1 --2- 172.21.15.59:0/3705434652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d1c0fe780 0x7f4d1c0fc270 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:12.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.018+0000 7f4d211f1700 1 -- 172.21.15.59:0/3705434652 >> 172.21.15.59:0/3705434652 conn(0x7f4d1c071c20 msgr2=0x7f4d1c1016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:12.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.018+0000 7f4d211f1700 1 -- 172.21.15.59:0/3705434652 shutdown_connections 2024-01-23T08:49:12.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:12.018+0000 7f4d211f1700 1 -- 172.21.15.59:0/3705434652 wait complete. 2024-01-23T08:49:13.314 INFO:teuthology.orchestra.run.smithi059.stdout:81604378648 2024-01-23T08:49:13.314 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-01-23T08:49:13.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.493+0000 7f375f086700 1 -- 172.21.15.59:0/2942843435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37580fb400 msgr2=0x7f37580fb850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:13.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.493+0000 7f375f086700 1 --2- 172.21.15.59:0/2942843435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37580fb400 0x7f37580fb850 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3744004940 tx=0x7f3744009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:13.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.494+0000 7f375f086700 1 -- 172.21.15.59:0/2942843435 shutdown_connections 2024-01-23T08:49:13.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.494+0000 7f375f086700 1 --2- 172.21.15.59:0/2942843435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37580fb400 0x7f37580fb850 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:13.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.494+0000 7f375f086700 1 --2- 172.21.15.59:0/2942843435 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3758102fd0 0x7f37581033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:13.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.494+0000 7f375f086700 1 -- 172.21.15.59:0/2942843435 >> 172.21.15.59:0/2942843435 conn(0x7f37580713f0 msgr2=0x7f37580717f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:13.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.494+0000 7f375f086700 1 -- 172.21.15.59:0/2942843435 shutdown_connections 2024-01-23T08:49:13.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.494+0000 7f375f086700 1 -- 172.21.15.59:0/2942843435 wait complete. 2024-01-23T08:49:13.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.494+0000 7f375f086700 1 Processor -- start 2024-01-23T08:49:13.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.494+0000 7f375f086700 1 -- start start 2024-01-23T08:49:13.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.494+0000 7f375f086700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37580fb400 0x7f3758113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:13.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.494+0000 7f375f086700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3758102fd0 0x7f3758113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:13.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.494+0000 7f375f086700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3758114510 con 0x7f37580fb400 2024-01-23T08:49:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.495+0000 7f375f086700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375810d840 con 0x7f3758102fd0 2024-01-23T08:49:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.495+0000 7f375ce22700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37580fb400 0x7f3758113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.495+0000 7f375ce22700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37580fb400 0x7f3758113980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36882/0 (socket says 172.21.15.59:36882) 2024-01-23T08:49:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.495+0000 7f375ce22700 1 -- 172.21.15.59:0/776321103 learned_addr learned my addr 172.21.15.59:0/776321103 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:49:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.495+0000 7f375ce22700 1 -- 172.21.15.59:0/776321103 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3758102fd0 msgr2=0x7f3758113ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.495+0000 7f375ce22700 1 --2- 172.21.15.59:0/776321103 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3758102fd0 0x7f3758113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.495+0000 7f375ce22700 1 -- 172.21.15.59:0/776321103 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3744005040 con 0x7f37580fb400 2024-01-23T08:49:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.495+0000 7f375ce22700 1 --2- 172.21.15.59:0/776321103 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37580fb400 0x7f3758113980 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f374c002a50 tx=0x7f374c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.495+0000 7f37557fa700 1 -- 172.21.15.59:0/776321103 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f374c007cc0 con 0x7f37580fb400 2024-01-23T08:49:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.495+0000 7f375f086700 1 -- 172.21.15.59:0/776321103 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f375810db20 con 0x7f37580fb400 2024-01-23T08:49:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.495+0000 7f37557fa700 1 -- 172.21.15.59:0/776321103 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f374c00f040 con 0x7f37580fb400 2024-01-23T08:49:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.495+0000 7f37557fa700 1 -- 172.21.15.59:0/776321103 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f374c00da70 con 0x7f37580fb400 2024-01-23T08:49:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.495+0000 7f375f086700 1 -- 172.21.15.59:0/776321103 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f375810e060 con 0x7f37580fb400 2024-01-23T08:49:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.496+0000 7f375f086700 1 -- 172.21.15.59:0/776321103 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3758105360 con 0x7f37580fb400 2024-01-23T08:49:13.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.497+0000 7f37557fa700 1 -- 172.21.15.59:0/776321103 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f374c00dc10 con 0x7f37580fb400 2024-01-23T08:49:13.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.497+0000 7f37557fa700 1 --2- 172.21.15.59:0/776321103 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f3748068f60 0x7f374806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:13.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.497+0000 7f3757fff700 1 --2- 172.21.15.59:0/776321103 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f3748068f60 0x7f374806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:13.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.497+0000 7f37557fa700 1 -- 172.21.15.59:0/776321103 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f374c0861b0 con 0x7f37580fb400 2024-01-23T08:49:13.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.498+0000 7f3757fff700 1 --2- 172.21.15.59:0/776321103 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f3748068f60 0x7f374806b410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3744004f90 tx=0x7f3744009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:13.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.502+0000 7f37557fa700 1 -- 172.21.15.59:0/776321103 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f374c0563d0 con 0x7f37580fb400 2024-01-23T08:49:13.503 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:49:13.636 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:13 smithi059 ceph-mon[28424]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:13.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.701+0000 7f375f086700 1 -- 172.21.15.59:0/776321103 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f375810ef60 con 0x7f37580fb400 2024-01-23T08:49:13.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.702+0000 7f37557fa700 1 -- 172.21.15.59:0/776321103 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f374c055f60 con 0x7f37580fb400 2024-01-23T08:49:13.699 INFO:teuthology.orchestra.run.smithi059.stdout:171798691850 2024-01-23T08:49:13.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.704+0000 7f375f086700 1 -- 172.21.15.59:0/776321103 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f3748068f60 msgr2=0x7f374806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:13.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.704+0000 7f375f086700 1 --2- 172.21.15.59:0/776321103 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f3748068f60 0x7f374806b410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3744004f90 tx=0x7f3744009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:13.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.704+0000 7f375f086700 1 -- 172.21.15.59:0/776321103 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37580fb400 msgr2=0x7f3758113980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:13.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.704+0000 7f375f086700 1 --2- 172.21.15.59:0/776321103 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37580fb400 0x7f3758113980 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f374c002a50 tx=0x7f374c0046b0 comp rx=0 tx=0).stop 2024-01-23T08:49:13.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.704+0000 7f375f086700 1 -- 172.21.15.59:0/776321103 shutdown_connections 2024-01-23T08:49:13.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.704+0000 7f375f086700 1 --2- 172.21.15.59:0/776321103 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f3748068f60 0x7f374806b410 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:13.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.704+0000 7f375f086700 1 --2- 172.21.15.59:0/776321103 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37580fb400 0x7f3758113980 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:13.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.704+0000 7f375f086700 1 --2- 172.21.15.59:0/776321103 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3758102fd0 0x7f3758113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:13.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.704+0000 7f375f086700 1 -- 172.21.15.59:0/776321103 >> 172.21.15.59:0/776321103 conn(0x7f37580713f0 msgr2=0x7f37580f9670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:13.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.704+0000 7f375f086700 1 -- 172.21.15.59:0/776321103 shutdown_connections 2024-01-23T08:49:13.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:13.704+0000 7f375f086700 1 -- 172.21.15.59:0/776321103 wait complete. 2024-01-23T08:49:13.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:13 smithi137 ceph-mon[35787]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:14.371 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691849 got 171798691850 for osd.5 2024-01-23T08:49:14.371 DEBUG:teuthology.parallel:result is None 2024-01-23T08:49:14.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.588+0000 7f02393e6700 1 -- 172.21.15.59:0/1382633645 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0234102fd0 msgr2=0x7f02341033a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:14.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.588+0000 7f02393e6700 1 --2- 172.21.15.59:0/1382633645 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0234102fd0 0x7f02341033a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f022c009ce0 tx=0x7f022c004d10 comp rx=0 tx=0).stop 2024-01-23T08:49:14.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.588+0000 7f02393e6700 1 -- 172.21.15.59:0/1382633645 shutdown_connections 2024-01-23T08:49:14.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.588+0000 7f02393e6700 1 --2- 172.21.15.59:0/1382633645 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02340fb400 0x7f02340fb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:14.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.588+0000 7f02393e6700 1 --2- 172.21.15.59:0/1382633645 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0234102fd0 0x7f02341033a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:14.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.588+0000 7f02393e6700 1 -- 172.21.15.59:0/1382633645 >> 172.21.15.59:0/1382633645 conn(0x7f02340713f0 msgr2=0x7f02340717f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:14.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.588+0000 7f02393e6700 1 -- 172.21.15.59:0/1382633645 shutdown_connections 2024-01-23T08:49:14.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.588+0000 7f02393e6700 1 -- 172.21.15.59:0/1382633645 wait complete. 2024-01-23T08:49:14.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.589+0000 7f02393e6700 1 Processor -- start 2024-01-23T08:49:14.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.589+0000 7f02393e6700 1 -- start start 2024-01-23T08:49:14.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.589+0000 7f02393e6700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02340fb400 0x7f02341139e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:14.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.589+0000 7f02393e6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0234102fd0 0x7f0234113f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:14.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.589+0000 7f02393e6700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0234114570 con 0x7f0234102fd0 2024-01-23T08:49:14.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.589+0000 7f02393e6700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f023410d8a0 con 0x7f02340fb400 2024-01-23T08:49:14.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.589+0000 7f02327fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0234102fd0 0x7f0234113f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:14.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.589+0000 7f02327fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0234102fd0 0x7f0234113f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36904/0 (socket says 172.21.15.59:36904) 2024-01-23T08:49:14.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.589+0000 7f02327fc700 1 -- 172.21.15.59:0/4262860288 learned_addr learned my addr 172.21.15.59:0/4262860288 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:49:14.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.590+0000 7f02327fc700 1 -- 172.21.15.59:0/4262860288 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02340fb400 msgr2=0x7f02341139e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:14.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.590+0000 7f0232ffd700 1 --2- 172.21.15.59:0/4262860288 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02340fb400 0x7f02341139e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:14.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.590+0000 7f02327fc700 1 --2- 172.21.15.59:0/4262860288 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02340fb400 0x7f02341139e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:14.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.590+0000 7f02327fc700 1 -- 172.21.15.59:0/4262860288 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022c009940 con 0x7f0234102fd0 2024-01-23T08:49:14.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.590+0000 7f02327fc700 1 --2- 172.21.15.59:0/4262860288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0234102fd0 0x7f0234113f20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f021c00c830 tx=0x7f021c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:14.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.590+0000 7f022b7fe700 1 -- 172.21.15.59:0/4262860288 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f021c00a760 con 0x7f0234102fd0 2024-01-23T08:49:14.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.590+0000 7f022b7fe700 1 -- 172.21.15.59:0/4262860288 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f021c00ad70 con 0x7f0234102fd0 2024-01-23T08:49:14.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.590+0000 7f02393e6700 1 -- 172.21.15.59:0/4262860288 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f023410db80 con 0x7f0234102fd0 2024-01-23T08:49:14.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.590+0000 7f022b7fe700 1 -- 172.21.15.59:0/4262860288 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f021c0113a0 con 0x7f0234102fd0 2024-01-23T08:49:14.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.590+0000 7f02393e6700 1 -- 172.21.15.59:0/4262860288 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f023410e0c0 con 0x7f0234102fd0 2024-01-23T08:49:14.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.591+0000 7f02393e6700 1 -- 172.21.15.59:0/4262860288 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f023410e5c0 con 0x7f0234102fd0 2024-01-23T08:49:14.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.592+0000 7f022b7fe700 1 -- 172.21.15.59:0/4262860288 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f021c012400 con 0x7f0234102fd0 2024-01-23T08:49:14.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.592+0000 7f022b7fe700 1 --2- 172.21.15.59:0/4262860288 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f0220068f60 0x7f022006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:14.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.593+0000 7f022b7fe700 1 -- 172.21.15.59:0/4262860288 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f021c086b70 con 0x7f0234102fd0 2024-01-23T08:49:14.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.593+0000 7f0232ffd700 1 --2- 172.21.15.59:0/4262860288 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f0220068f60 0x7f022006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:14.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.593+0000 7f022b7fe700 1 --2- 172.21.15.59:0/4262860288 >> [v2:172.21.15.59:6802/2564662034,v1:172.21.15.59:6803/2564662034] conn(0x7f022006ec30 0x7f0220071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:14.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.593+0000 7f022b7fe700 1 -- 172.21.15.59:0/4262860288 --> [v2:172.21.15.59:6802/2564662034,v1:172.21.15.59:6803/2564662034] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02200716f0 con 0x7f022006ec30 2024-01-23T08:49:14.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.593+0000 7f022b7fe700 1 -- 172.21.15.59:0/4262860288 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f021c08b220 con 0x7f0234102fd0 2024-01-23T08:49:14.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.593+0000 7f02337fe700 1 --2- 172.21.15.59:0/4262860288 >> [v2:172.21.15.59:6802/2564662034,v1:172.21.15.59:6803/2564662034] conn(0x7f022006ec30 0x7f0220071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:14.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.593+0000 7f0232ffd700 1 --2- 172.21.15.59:0/4262860288 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f0220068f60 0x7f022006b410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f022c015040 tx=0x7f022c004d10 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:14.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.593+0000 7f02337fe700 1 --2- 172.21.15.59:0/4262860288 >> [v2:172.21.15.59:6802/2564662034,v1:172.21.15.59:6803/2564662034] conn(0x7f022006ec30 0x7f0220071040 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:14.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.594+0000 7f022b7fe700 1 -- 172.21.15.59:0/4262860288 <== osd.0 v2:172.21.15.59:6802/2564662034 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f02200716f0 con 0x7f022006ec30 2024-01-23T08:49:14.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.610+0000 7f02393e6700 1 -- 172.21.15.59:0/4262860288 --> [v2:172.21.15.59:6802/2564662034,v1:172.21.15.59:6803/2564662034] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f023405e4a0 con 0x7f022006ec30 2024-01-23T08:49:14.607 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:14 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/776321103' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-01-23T08:49:14.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.610+0000 7f022b7fe700 1 -- 172.21.15.59:0/4262860288 <== osd.0 v2:172.21.15.59:6802/2564662034 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f023405e4a0 con 0x7f022006ec30 2024-01-23T08:49:14.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.611+0000 7f02393e6700 1 -- 172.21.15.59:0/4262860288 >> [v2:172.21.15.59:6802/2564662034,v1:172.21.15.59:6803/2564662034] conn(0x7f022006ec30 msgr2=0x7f0220071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:14.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.611+0000 7f02393e6700 1 --2- 172.21.15.59:0/4262860288 >> [v2:172.21.15.59:6802/2564662034,v1:172.21.15.59:6803/2564662034] conn(0x7f022006ec30 0x7f0220071040 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:14.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.611+0000 7f02393e6700 1 -- 172.21.15.59:0/4262860288 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f0220068f60 msgr2=0x7f022006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:14.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.611+0000 7f02393e6700 1 --2- 172.21.15.59:0/4262860288 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f0220068f60 0x7f022006b410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f022c015040 tx=0x7f022c004d10 comp rx=0 tx=0).stop 2024-01-23T08:49:14.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.611+0000 7f02393e6700 1 -- 172.21.15.59:0/4262860288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0234102fd0 msgr2=0x7f0234113f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:14.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.611+0000 7f02393e6700 1 --2- 172.21.15.59:0/4262860288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0234102fd0 0x7f0234113f20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f021c00c830 tx=0x7f021c004470 comp rx=0 tx=0).stop 2024-01-23T08:49:14.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.611+0000 7f02393e6700 1 -- 172.21.15.59:0/4262860288 shutdown_connections 2024-01-23T08:49:14.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.611+0000 7f02393e6700 1 --2- 172.21.15.59:0/4262860288 >> [v2:172.21.15.59:6802/2564662034,v1:172.21.15.59:6803/2564662034] conn(0x7f022006ec30 0x7f0220071040 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:14.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.611+0000 7f02393e6700 1 --2- 172.21.15.59:0/4262860288 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f0220068f60 0x7f022006b410 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:14.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.611+0000 7f02393e6700 1 --2- 172.21.15.59:0/4262860288 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02340fb400 0x7f02341139e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:14.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.611+0000 7f02393e6700 1 --2- 172.21.15.59:0/4262860288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0234102fd0 0x7f0234113f20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:14.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.611+0000 7f02393e6700 1 -- 172.21.15.59:0/4262860288 >> 172.21.15.59:0/4262860288 conn(0x7f02340713f0 msgr2=0x7f02340fa4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:14.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.611+0000 7f02393e6700 1 -- 172.21.15.59:0/4262860288 shutdown_connections 2024-01-23T08:49:14.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:14.611+0000 7f02393e6700 1 -- 172.21.15.59:0/4262860288 wait complete. 2024-01-23T08:49:14.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:14 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/776321103' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-01-23T08:49:15.413 INFO:teuthology.orchestra.run.smithi059.stdout:150323855375 2024-01-23T08:49:15.413 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-01-23T08:49:15.718 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:15 smithi059 ceph-mon[28424]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:15.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:15 smithi137 ceph-mon[35787]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:16.051 INFO:teuthology.orchestra.run.smithi059.stdout:38654705701 2024-01-23T08:49:16.051 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-01-23T08:49:16.966 INFO:teuthology.orchestra.run.smithi059.stdout:115964117012 2024-01-23T08:49:16.966 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-01-23T08:49:17.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.431+0000 7f6defac3700 1 -- 172.21.15.59:0/1396392028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de8101cf0 msgr2=0x7f6de81020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:17.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.431+0000 7f6defac3700 1 --2- 172.21.15.59:0/1396392028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de8101cf0 0x7f6de81020c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6de4004940 tx=0x7f6de4009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:17.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.431+0000 7f6defac3700 1 -- 172.21.15.59:0/1396392028 shutdown_connections 2024-01-23T08:49:17.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.431+0000 7f6defac3700 1 --2- 172.21.15.59:0/1396392028 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6de8102690 0x7f6de80fbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:17.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.431+0000 7f6defac3700 1 --2- 172.21.15.59:0/1396392028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de8101cf0 0x7f6de81020c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:17.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.431+0000 7f6defac3700 1 -- 172.21.15.59:0/1396392028 >> 172.21.15.59:0/1396392028 conn(0x7f6de80f78a0 msgr2=0x7f6de80f9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:17.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.431+0000 7f6defac3700 1 -- 172.21.15.59:0/1396392028 shutdown_connections 2024-01-23T08:49:17.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.431+0000 7f6defac3700 1 -- 172.21.15.59:0/1396392028 wait complete. 2024-01-23T08:49:17.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.431+0000 7f6defac3700 1 Processor -- start 2024-01-23T08:49:17.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.431+0000 7f6defac3700 1 -- start start 2024-01-23T08:49:17.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.431+0000 7f6defac3700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6de8101cf0 0x7f6de8102b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:17.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.431+0000 7f6defac3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de8102690 0x7f6de8103090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:17.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.431+0000 7f6defac3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de8103690 con 0x7f6de8102690 2024-01-23T08:49:17.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.432+0000 7f6defac3700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de8103800 con 0x7f6de8101cf0 2024-01-23T08:49:17.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.432+0000 7f6ded05e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de8102690 0x7f6de8103090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:17.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.432+0000 7f6ded05e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de8102690 0x7f6de8103090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36924/0 (socket says 172.21.15.59:36924) 2024-01-23T08:49:17.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.432+0000 7f6ded05e700 1 -- 172.21.15.59:0/1441955041 learned_addr learned my addr 172.21.15.59:0/1441955041 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:49:17.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.432+0000 7f6ded85f700 1 --2- 172.21.15.59:0/1441955041 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6de8101cf0 0x7f6de8102b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:17.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.432+0000 7f6ded05e700 1 -- 172.21.15.59:0/1441955041 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6de8101cf0 msgr2=0x7f6de8102b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:17.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.432+0000 7f6ded05e700 1 --2- 172.21.15.59:0/1441955041 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6de8101cf0 0x7f6de8102b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:17.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.432+0000 7f6ded05e700 1 -- 172.21.15.59:0/1441955041 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de4005040 con 0x7f6de8102690 2024-01-23T08:49:17.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.432+0000 7f6ded05e700 1 --2- 172.21.15.59:0/1441955041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de8102690 0x7f6de8103090 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8002a50 tx=0x7f6dd8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:17.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.432+0000 7f6dde7fc700 1 -- 172.21.15.59:0/1441955041 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dd8007cc0 con 0x7f6de8102690 2024-01-23T08:49:17.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.432+0000 7f6dde7fc700 1 -- 172.21.15.59:0/1441955041 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6dd800f040 con 0x7f6de8102690 2024-01-23T08:49:17.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.432+0000 7f6dde7fc700 1 -- 172.21.15.59:0/1441955041 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dd800d990 con 0x7f6de8102690 2024-01-23T08:49:17.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.432+0000 7f6defac3700 1 -- 172.21.15.59:0/1441955041 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de8119a90 con 0x7f6de8102690 2024-01-23T08:49:17.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.432+0000 7f6defac3700 1 -- 172.21.15.59:0/1441955041 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de8119f70 con 0x7f6de8102690 2024-01-23T08:49:17.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.433+0000 7f6defac3700 1 -- 172.21.15.59:0/1441955041 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de80fb4a0 con 0x7f6de8102690 2024-01-23T08:49:17.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.434+0000 7f6dde7fc700 1 -- 172.21.15.59:0/1441955041 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f6dd8009710 con 0x7f6de8102690 2024-01-23T08:49:17.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.434+0000 7f6dde7fc700 1 --2- 172.21.15.59:0/1441955041 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f6dd4068f60 0x7f6dd406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:17.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.434+0000 7f6dde7fc700 1 -- 172.21.15.59:0/1441955041 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6dd80867c0 con 0x7f6de8102690 2024-01-23T08:49:17.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.434+0000 7f6ded85f700 1 --2- 172.21.15.59:0/1441955041 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f6dd4068f60 0x7f6dd406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:17.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.435+0000 7f6ded85f700 1 --2- 172.21.15.59:0/1441955041 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f6dd4068f60 0x7f6dd406b410 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6de4004f90 tx=0x7f6de4009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:17.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.439+0000 7f6dde7fc700 1 -- 172.21.15.59:0/1441955041 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6dd8052fd0 con 0x7f6de8102690 2024-01-23T08:49:17.448 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:49:17.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:17 smithi059 ceph-mon[28424]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:17.503 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:49:17.503 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:49:17.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.656+0000 7f6defac3700 1 -- 172.21.15.59:0/1441955041 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f6de8108cb0 con 0x7f6de8102690 2024-01-23T08:49:17.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.657+0000 7f6dde7fc700 1 -- 172.21.15.59:0/1441955041 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6dd80565f0 con 0x7f6de8102690 2024-01-23T08:49:17.654 INFO:teuthology.orchestra.run.smithi059.stdout:60129542175 2024-01-23T08:49:17.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.659+0000 7f6defac3700 1 -- 172.21.15.59:0/1441955041 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f6dd4068f60 msgr2=0x7f6dd406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:17.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.659+0000 7f6defac3700 1 --2- 172.21.15.59:0/1441955041 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f6dd4068f60 0x7f6dd406b410 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6de4004f90 tx=0x7f6de4009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:17.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.659+0000 7f6defac3700 1 -- 172.21.15.59:0/1441955041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de8102690 msgr2=0x7f6de8103090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:17.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.659+0000 7f6defac3700 1 --2- 172.21.15.59:0/1441955041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de8102690 0x7f6de8103090 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8002a50 tx=0x7f6dd8004600 comp rx=0 tx=0).stop 2024-01-23T08:49:17.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.659+0000 7f6defac3700 1 -- 172.21.15.59:0/1441955041 shutdown_connections 2024-01-23T08:49:17.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.659+0000 7f6defac3700 1 --2- 172.21.15.59:0/1441955041 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f6dd4068f60 0x7f6dd406b410 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:17.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.659+0000 7f6defac3700 1 --2- 172.21.15.59:0/1441955041 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6de8101cf0 0x7f6de8102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:17.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.659+0000 7f6defac3700 1 --2- 172.21.15.59:0/1441955041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de8102690 0x7f6de8103090 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:17.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.659+0000 7f6defac3700 1 -- 172.21.15.59:0/1441955041 >> 172.21.15.59:0/1441955041 conn(0x7f6de80f78a0 msgr2=0x7f6de8065f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:17.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.659+0000 7f6defac3700 1 -- 172.21.15.59:0/1441955041 shutdown_connections 2024-01-23T08:49:17.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:17.659+0000 7f6defac3700 1 -- 172.21.15.59:0/1441955041 wait complete. 2024-01-23T08:49:17.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:17 smithi137 ceph-mon[35787]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:18.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:18 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/1441955041' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-01-23T08:49:18.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:18 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/1441955041' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-01-23T08:49:18.822 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:49:18.851 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542172 got 60129542175 for osd.1 2024-01-23T08:49:18.851 DEBUG:teuthology.parallel:result is None 2024-01-23T08:49:19.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:19 smithi059 ceph-mon[28424]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:19.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:19 smithi137 ceph-mon[35787]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:20.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.642+0000 7f6244c18700 1 -- 172.21.15.59:0/1680980195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6240103780 msgr2=0x7f62400fcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:20.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.642+0000 7f6244c18700 1 --2- 172.21.15.59:0/1680980195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6240103780 0x7f62400fcdf0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6228004940 tx=0x7f6228009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:20.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.642+0000 7f6244c18700 1 -- 172.21.15.59:0/1680980195 shutdown_connections 2024-01-23T08:49:20.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.642+0000 7f6244c18700 1 --2- 172.21.15.59:0/1680980195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6240103780 0x7f62400fcdf0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:20.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.642+0000 7f6244c18700 1 --2- 172.21.15.59:0/1680980195 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6240102de0 0x7f62401031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:20.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.642+0000 7f6244c18700 1 -- 172.21.15.59:0/1680980195 >> 172.21.15.59:0/1680980195 conn(0x7f62400f8970 msgr2=0x7f62400fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:20.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.642+0000 7f6244c18700 1 -- 172.21.15.59:0/1680980195 shutdown_connections 2024-01-23T08:49:20.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.642+0000 7f6244c18700 1 -- 172.21.15.59:0/1680980195 wait complete. 2024-01-23T08:49:20.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.643+0000 7f6244c18700 1 Processor -- start 2024-01-23T08:49:20.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.643+0000 7f6244c18700 1 -- start start 2024-01-23T08:49:20.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.643+0000 7f6244c18700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6240102de0 0x7f6240113850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:20.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.643+0000 7f6244c18700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6240103780 0x7f6240113d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:20.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.643+0000 7f6244c18700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6240114470 con 0x7f6240102de0 2024-01-23T08:49:20.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.643+0000 7f6244c18700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624010d710 con 0x7f6240103780 2024-01-23T08:49:20.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.643+0000 7f623e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6240102de0 0x7f6240113850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:20.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.643+0000 7f623e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6240102de0 0x7f6240113850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37998/0 (socket says 172.21.15.59:37998) 2024-01-23T08:49:20.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.643+0000 7f623e59c700 1 -- 172.21.15.59:0/98777566 learned_addr learned my addr 172.21.15.59:0/98777566 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:49:20.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.643+0000 7f623dd9b700 1 --2- 172.21.15.59:0/98777566 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6240103780 0x7f6240113d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:20.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.643+0000 7f623e59c700 1 -- 172.21.15.59:0/98777566 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6240103780 msgr2=0x7f6240113d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:20.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.643+0000 7f623e59c700 1 --2- 172.21.15.59:0/98777566 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6240103780 0x7f6240113d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:20.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.643+0000 7f623e59c700 1 -- 172.21.15.59:0/98777566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6228005040 con 0x7f6240102de0 2024-01-23T08:49:20.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.643+0000 7f623e59c700 1 --2- 172.21.15.59:0/98777566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6240102de0 0x7f6240113850 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f623000bbb0 tx=0x7f6230009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:20.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.644+0000 7f6236ffd700 1 -- 172.21.15.59:0/98777566 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62300092e0 con 0x7f6240102de0 2024-01-23T08:49:20.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.644+0000 7f6236ffd700 1 -- 172.21.15.59:0/98777566 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f623000cbb0 con 0x7f6240102de0 2024-01-23T08:49:20.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.644+0000 7f6244c18700 1 -- 172.21.15.59:0/98777566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f624010d9f0 con 0x7f6240102de0 2024-01-23T08:49:20.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.644+0000 7f6236ffd700 1 -- 172.21.15.59:0/98777566 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6230015d70 con 0x7f6240102de0 2024-01-23T08:49:20.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.644+0000 7f6244c18700 1 -- 172.21.15.59:0/98777566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f624010df30 con 0x7f6240102de0 2024-01-23T08:49:20.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.645+0000 7f6244c18700 1 -- 172.21.15.59:0/98777566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62401000f0 con 0x7f6240102de0 2024-01-23T08:49:20.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.645+0000 7f6236ffd700 1 -- 172.21.15.59:0/98777566 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f6230012070 con 0x7f6240102de0 2024-01-23T08:49:20.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.646+0000 7f6236ffd700 1 --2- 172.21.15.59:0/98777566 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f622c068f60 0x7f622c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:20.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.646+0000 7f6236ffd700 1 -- 172.21.15.59:0/98777566 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6230086840 con 0x7f6240102de0 2024-01-23T08:49:20.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.646+0000 7f623dd9b700 1 --2- 172.21.15.59:0/98777566 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f622c068f60 0x7f622c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:20.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.646+0000 7f623dd9b700 1 --2- 172.21.15.59:0/98777566 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f622c068f60 0x7f622c06b410 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f624010f080 tx=0x7f6228009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:20.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.650+0000 7f6236ffd700 1 -- 172.21.15.59:0/98777566 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6230056a60 con 0x7f6240102de0 2024-01-23T08:49:20.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.848+0000 7f6244c18700 1 -- 172.21.15.59:0/98777566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f624004bd60 con 0x7f6240102de0 2024-01-23T08:49:20.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.848+0000 7f6236ffd700 1 -- 172.21.15.59:0/98777566 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f62300565f0 con 0x7f6240102de0 2024-01-23T08:49:20.846 INFO:teuthology.orchestra.run.smithi059.stdout:81604378650 2024-01-23T08:49:20.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.850+0000 7f6244c18700 1 -- 172.21.15.59:0/98777566 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f622c068f60 msgr2=0x7f622c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:20.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.850+0000 7f6244c18700 1 --2- 172.21.15.59:0/98777566 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f622c068f60 0x7f622c06b410 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f624010f080 tx=0x7f6228009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:20.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.850+0000 7f6244c18700 1 -- 172.21.15.59:0/98777566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6240102de0 msgr2=0x7f6240113850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:20.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.850+0000 7f6244c18700 1 --2- 172.21.15.59:0/98777566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6240102de0 0x7f6240113850 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f623000bbb0 tx=0x7f6230009840 comp rx=0 tx=0).stop 2024-01-23T08:49:20.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.850+0000 7f6244c18700 1 -- 172.21.15.59:0/98777566 shutdown_connections 2024-01-23T08:49:20.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.850+0000 7f6244c18700 1 --2- 172.21.15.59:0/98777566 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f622c068f60 0x7f622c06b410 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:20.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.850+0000 7f6244c18700 1 --2- 172.21.15.59:0/98777566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6240102de0 0x7f6240113850 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:20.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.850+0000 7f6244c18700 1 --2- 172.21.15.59:0/98777566 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6240103780 0x7f6240113d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:20.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.850+0000 7f6244c18700 1 -- 172.21.15.59:0/98777566 >> 172.21.15.59:0/98777566 conn(0x7f62400f8970 msgr2=0x7f62400fa390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:20.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.851+0000 7f6244c18700 1 -- 172.21.15.59:0/98777566 shutdown_connections 2024-01-23T08:49:20.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:20.851+0000 7f6244c18700 1 -- 172.21.15.59:0/98777566 wait complete. 2024-01-23T08:49:21.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:21 smithi059 ceph-mon[28424]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:21.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:21 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/98777566' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-01-23T08:49:21.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:21 smithi137 ceph-mon[35787]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:21.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:21 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/98777566' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-01-23T08:49:23.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:23 smithi059 ceph-mon[28424]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:23.727 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378648 got 81604378650 for osd.2 2024-01-23T08:49:23.727 DEBUG:teuthology.parallel:result is None 2024-01-23T08:49:23.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:23 smithi137 ceph-mon[35787]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:24.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.191+0000 7fedfb369700 1 -- 172.21.15.59:0/3003695537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedf4103770 msgr2=0x7fedf40fcde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:24.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.191+0000 7fedfb369700 1 --2- 172.21.15.59:0/3003695537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedf4103770 0x7fedf40fcde0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fede4004940 tx=0x7fede4009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:24.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.191+0000 7fedfb369700 1 -- 172.21.15.59:0/3003695537 shutdown_connections 2024-01-23T08:49:24.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.191+0000 7fedfb369700 1 --2- 172.21.15.59:0/3003695537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedf4103770 0x7fedf40fcde0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:24.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.191+0000 7fedfb369700 1 --2- 172.21.15.59:0/3003695537 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedf4102dd0 0x7fedf41031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:24.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.191+0000 7fedfb369700 1 -- 172.21.15.59:0/3003695537 >> 172.21.15.59:0/3003695537 conn(0x7fedf40f8980 msgr2=0x7fedf40fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:24.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.191+0000 7fedfb369700 1 -- 172.21.15.59:0/3003695537 shutdown_connections 2024-01-23T08:49:24.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.192+0000 7fedfb369700 1 -- 172.21.15.59:0/3003695537 wait complete. 2024-01-23T08:49:24.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.192+0000 7fedfb369700 1 Processor -- start 2024-01-23T08:49:24.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.192+0000 7fedfb369700 1 -- start start 2024-01-23T08:49:24.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.192+0000 7fedfb369700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedf4102dd0 0x7fedf4113840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:24.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.192+0000 7fedfb369700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedf4103770 0x7fedf4113d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:24.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.192+0000 7fedfb369700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf4114460 con 0x7fedf4103770 2024-01-23T08:49:24.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.192+0000 7fedfb369700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf410d700 con 0x7fedf4102dd0 2024-01-23T08:49:24.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.192+0000 7fedf8904700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedf4103770 0x7fedf4113d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:24.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.192+0000 7fedf8904700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedf4103770 0x7fedf4113d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38024/0 (socket says 172.21.15.59:38024) 2024-01-23T08:49:24.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.192+0000 7fedf8904700 1 -- 172.21.15.59:0/1549261489 learned_addr learned my addr 172.21.15.59:0/1549261489 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:49:24.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.193+0000 7fedf9105700 1 --2- 172.21.15.59:0/1549261489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedf4102dd0 0x7fedf4113840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:24.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.193+0000 7fedf8904700 1 -- 172.21.15.59:0/1549261489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedf4102dd0 msgr2=0x7fedf4113840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:24.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.193+0000 7fedf8904700 1 --2- 172.21.15.59:0/1549261489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedf4102dd0 0x7fedf4113840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:24.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.193+0000 7fedf8904700 1 -- 172.21.15.59:0/1549261489 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede4005040 con 0x7fedf4103770 2024-01-23T08:49:24.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.193+0000 7fedf9105700 1 --2- 172.21.15.59:0/1549261489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedf4102dd0 0x7fedf4113840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-23T08:49:24.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.193+0000 7fedf8904700 1 --2- 172.21.15.59:0/1549261489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedf4103770 0x7fedf4113d80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fede4009fd0 tx=0x7fede4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:24.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.193+0000 7fede9ffb700 1 -- 172.21.15.59:0/1549261489 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fede400e040 con 0x7fedf4103770 2024-01-23T08:49:24.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.193+0000 7fede9ffb700 1 -- 172.21.15.59:0/1549261489 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fede400bc50 con 0x7fedf4103770 2024-01-23T08:49:24.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.193+0000 7fedfb369700 1 -- 172.21.15.59:0/1549261489 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf410d980 con 0x7fedf4103770 2024-01-23T08:49:24.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.193+0000 7fede9ffb700 1 -- 172.21.15.59:0/1549261489 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fede401c3f0 con 0x7fedf4103770 2024-01-23T08:49:24.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.193+0000 7fedfb369700 1 -- 172.21.15.59:0/1549261489 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf410dec0 con 0x7fedf4103770 2024-01-23T08:49:24.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.194+0000 7fedfb369700 1 -- 172.21.15.59:0/1549261489 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf41000e0 con 0x7fedf4103770 2024-01-23T08:49:24.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.195+0000 7fede9ffb700 1 -- 172.21.15.59:0/1549261489 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fede4013420 con 0x7fedf4103770 2024-01-23T08:49:24.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.195+0000 7fede9ffb700 1 --2- 172.21.15.59:0/1549261489 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fede0068f60 0x7fede006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:24.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.195+0000 7fede9ffb700 1 -- 172.21.15.59:0/1549261489 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fede4087e20 con 0x7fedf4103770 2024-01-23T08:49:24.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.195+0000 7fedf9105700 1 --2- 172.21.15.59:0/1549261489 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fede0068f60 0x7fede006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:24.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.196+0000 7fedf9105700 1 --2- 172.21.15.59:0/1549261489 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fede0068f60 0x7fede006b410 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fedf0006fd0 tx=0x7fedf0008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:24.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.200+0000 7fede9ffb700 1 -- 172.21.15.59:0/1549261489 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fede40580c0 con 0x7fedf4103770 2024-01-23T08:49:24.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.397+0000 7fedfb369700 1 -- 172.21.15.59:0/1549261489 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fedf404bd60 con 0x7fedf4103770 2024-01-23T08:49:24.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.397+0000 7fede9ffb700 1 -- 172.21.15.59:0/1549261489 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fede4057c50 con 0x7fedf4103770 2024-01-23T08:49:24.395 INFO:teuthology.orchestra.run.smithi059.stdout:115964117014 2024-01-23T08:49:24.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.399+0000 7fedfb369700 1 -- 172.21.15.59:0/1549261489 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fede0068f60 msgr2=0x7fede006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:24.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.399+0000 7fedfb369700 1 --2- 172.21.15.59:0/1549261489 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fede0068f60 0x7fede006b410 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fedf0006fd0 tx=0x7fedf0008040 comp rx=0 tx=0).stop 2024-01-23T08:49:24.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.399+0000 7fedfb369700 1 -- 172.21.15.59:0/1549261489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedf4103770 msgr2=0x7fedf4113d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:24.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.399+0000 7fedfb369700 1 --2- 172.21.15.59:0/1549261489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedf4103770 0x7fedf4113d80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fede4009fd0 tx=0x7fede4009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:24.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.399+0000 7fedfb369700 1 -- 172.21.15.59:0/1549261489 shutdown_connections 2024-01-23T08:49:24.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.399+0000 7fedfb369700 1 --2- 172.21.15.59:0/1549261489 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fede0068f60 0x7fede006b410 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:24.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.399+0000 7fedfb369700 1 --2- 172.21.15.59:0/1549261489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedf4102dd0 0x7fedf4113840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:24.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.399+0000 7fedfb369700 1 --2- 172.21.15.59:0/1549261489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedf4103770 0x7fedf4113d80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:24.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.399+0000 7fedfb369700 1 -- 172.21.15.59:0/1549261489 >> 172.21.15.59:0/1549261489 conn(0x7fedf40f8980 msgr2=0x7fedf40fa470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:24.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.399+0000 7fedfb369700 1 -- 172.21.15.59:0/1549261489 shutdown_connections 2024-01-23T08:49:24.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:24.399+0000 7fedfb369700 1 -- 172.21.15.59:0/1549261489 wait complete. 2024-01-23T08:49:24.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:24 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/1549261489' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-01-23T08:49:24.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:24 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/1549261489' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-01-23T08:49:25.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.033+0000 7fa74c42b700 1 -- 172.21.15.59:0/636676626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa744102dd0 msgr2=0x7fa7441031a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:25.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.033+0000 7fa74c42b700 1 --2- 172.21.15.59:0/636676626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa744102dd0 0x7fa7441031a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa734009ce0 tx=0x7fa734004d10 comp rx=0 tx=0).stop 2024-01-23T08:49:25.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.033+0000 7fa74c42b700 1 -- 172.21.15.59:0/636676626 shutdown_connections 2024-01-23T08:49:25.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.033+0000 7fa74c42b700 1 --2- 172.21.15.59:0/636676626 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa744103770 0x7fa7440fcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:25.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.033+0000 7fa74c42b700 1 --2- 172.21.15.59:0/636676626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa744102dd0 0x7fa7441031a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:25.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.033+0000 7fa74c42b700 1 -- 172.21.15.59:0/636676626 >> 172.21.15.59:0/636676626 conn(0x7fa7440f8980 msgr2=0x7fa7440fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:25.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.033+0000 7fa74c42b700 1 -- 172.21.15.59:0/636676626 shutdown_connections 2024-01-23T08:49:25.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.033+0000 7fa74c42b700 1 -- 172.21.15.59:0/636676626 wait complete. 2024-01-23T08:49:25.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.033+0000 7fa74c42b700 1 Processor -- start 2024-01-23T08:49:25.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.033+0000 7fa74c42b700 1 -- start start 2024-01-23T08:49:25.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.034+0000 7fa74c42b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa744102dd0 0x7fa744113a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:25.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.034+0000 7fa74c42b700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa744103770 0x7fa744113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:25.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.034+0000 7fa74c42b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa744114620 con 0x7fa744102dd0 2024-01-23T08:49:25.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.034+0000 7fa74c42b700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74410d8c0 con 0x7fa744103770 2024-01-23T08:49:25.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.034+0000 7fa74a1c7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa744102dd0 0x7fa744113a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:25.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.034+0000 7fa74a1c7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa744102dd0 0x7fa744113a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38050/0 (socket says 172.21.15.59:38050) 2024-01-23T08:49:25.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.034+0000 7fa74a1c7700 1 -- 172.21.15.59:0/1217029341 learned_addr learned my addr 172.21.15.59:0/1217029341 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:49:25.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.034+0000 7fa7499c6700 1 --2- 172.21.15.59:0/1217029341 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa744103770 0x7fa744113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:25.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.034+0000 7fa74a1c7700 1 -- 172.21.15.59:0/1217029341 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa744103770 msgr2=0x7fa744113f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:25.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.034+0000 7fa74a1c7700 1 --2- 172.21.15.59:0/1217029341 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa744103770 0x7fa744113f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:25.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.034+0000 7fa74a1c7700 1 -- 172.21.15.59:0/1217029341 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa734009940 con 0x7fa744102dd0 2024-01-23T08:49:25.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.034+0000 7fa74a1c7700 1 --2- 172.21.15.59:0/1217029341 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa744102dd0 0x7fa744113a00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa73400bee0 tx=0x7fa734004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:25.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.034+0000 7fa742ffd700 1 -- 172.21.15.59:0/1217029341 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa73400e040 con 0x7fa744102dd0 2024-01-23T08:49:25.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.034+0000 7fa74c42b700 1 -- 172.21.15.59:0/1217029341 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa74410db40 con 0x7fa744102dd0 2024-01-23T08:49:25.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.034+0000 7fa742ffd700 1 -- 172.21.15.59:0/1217029341 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa734008f00 con 0x7fa744102dd0 2024-01-23T08:49:25.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.034+0000 7fa742ffd700 1 -- 172.21.15.59:0/1217029341 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa73401c710 con 0x7fa744102dd0 2024-01-23T08:49:25.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.035+0000 7fa74c42b700 1 -- 172.21.15.59:0/1217029341 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa74410e080 con 0x7fa744102dd0 2024-01-23T08:49:25.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.035+0000 7fa74c42b700 1 -- 172.21.15.59:0/1217029341 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7440fc580 con 0x7fa744102dd0 2024-01-23T08:49:25.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.036+0000 7fa742ffd700 1 -- 172.21.15.59:0/1217029341 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fa734012b50 con 0x7fa744102dd0 2024-01-23T08:49:25.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.036+0000 7fa742ffd700 1 --2- 172.21.15.59:0/1217029341 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fa72c068f60 0x7fa72c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:25.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.036+0000 7fa742ffd700 1 -- 172.21.15.59:0/1217029341 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa734088740 con 0x7fa744102dd0 2024-01-23T08:49:25.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.036+0000 7fa7499c6700 1 --2- 172.21.15.59:0/1217029341 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fa72c068f60 0x7fa72c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:25.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.037+0000 7fa7499c6700 1 --2- 172.21.15.59:0/1217029341 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fa72c068f60 0x7fa72c06b410 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa738006fd0 tx=0x7fa738008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:25.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.041+0000 7fa742ffd700 1 -- 172.21.15.59:0/1217029341 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa734058960 con 0x7fa744102dd0 2024-01-23T08:49:25.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.235+0000 7fa74c42b700 1 -- 172.21.15.59:0/1217029341 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fa74410efe0 con 0x7fa744102dd0 2024-01-23T08:49:25.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.236+0000 7fa742ffd700 1 -- 172.21.15.59:0/1217029341 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa7340584f0 con 0x7fa744102dd0 2024-01-23T08:49:25.233 INFO:teuthology.orchestra.run.smithi059.stdout:38654705703 2024-01-23T08:49:25.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.238+0000 7fa74c42b700 1 -- 172.21.15.59:0/1217029341 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fa72c068f60 msgr2=0x7fa72c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:25.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.238+0000 7fa74c42b700 1 --2- 172.21.15.59:0/1217029341 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fa72c068f60 0x7fa72c06b410 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa738006fd0 tx=0x7fa738008040 comp rx=0 tx=0).stop 2024-01-23T08:49:25.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.238+0000 7fa74c42b700 1 -- 172.21.15.59:0/1217029341 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa744102dd0 msgr2=0x7fa744113a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:25.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.238+0000 7fa74c42b700 1 --2- 172.21.15.59:0/1217029341 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa744102dd0 0x7fa744113a00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa73400bee0 tx=0x7fa734004d10 comp rx=0 tx=0).stop 2024-01-23T08:49:25.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.238+0000 7fa74c42b700 1 -- 172.21.15.59:0/1217029341 shutdown_connections 2024-01-23T08:49:25.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.238+0000 7fa74c42b700 1 --2- 172.21.15.59:0/1217029341 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fa72c068f60 0x7fa72c06b410 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:25.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.238+0000 7fa74c42b700 1 --2- 172.21.15.59:0/1217029341 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa744102dd0 0x7fa744113a00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:25.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.238+0000 7fa74c42b700 1 --2- 172.21.15.59:0/1217029341 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa744103770 0x7fa744113f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:25.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.238+0000 7fa74c42b700 1 -- 172.21.15.59:0/1217029341 >> 172.21.15.59:0/1217029341 conn(0x7fa7440f8980 msgr2=0x7fa7440fa400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:25.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.238+0000 7fa74c42b700 1 -- 172.21.15.59:0/1217029341 shutdown_connections 2024-01-23T08:49:25.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.238+0000 7fa74c42b700 1 -- 172.21.15.59:0/1217029341 wait complete. 2024-01-23T08:49:25.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.260+0000 7f90fc30d700 1 -- 172.21.15.59:0/1968569253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90f4103770 msgr2=0x7f90f40fcde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:25.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.260+0000 7f90fc30d700 1 --2- 172.21.15.59:0/1968569253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90f4103770 0x7f90f40fcde0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f90e4004940 tx=0x7f90e4009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:25.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.260+0000 7f90fc30d700 1 -- 172.21.15.59:0/1968569253 shutdown_connections 2024-01-23T08:49:25.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.260+0000 7f90fc30d700 1 --2- 172.21.15.59:0/1968569253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90f4103770 0x7f90f40fcde0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:25.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.260+0000 7f90fc30d700 1 --2- 172.21.15.59:0/1968569253 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90f4102dd0 0x7f90f41031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:25.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.260+0000 7f90fc30d700 1 -- 172.21.15.59:0/1968569253 >> 172.21.15.59:0/1968569253 conn(0x7f90f40f8980 msgr2=0x7f90f40fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:25.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.260+0000 7f90fc30d700 1 -- 172.21.15.59:0/1968569253 shutdown_connections 2024-01-23T08:49:25.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.260+0000 7f90fc30d700 1 -- 172.21.15.59:0/1968569253 wait complete. 2024-01-23T08:49:25.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.261+0000 7f90fc30d700 1 Processor -- start 2024-01-23T08:49:25.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.261+0000 7f90fc30d700 1 -- start start 2024-01-23T08:49:25.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.261+0000 7f90fc30d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90f4102dd0 0x7f90f41138d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:25.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.261+0000 7f90fc30d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90f4103770 0x7f90f4113e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:25.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.261+0000 7f90fc30d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f4114460 con 0x7f90f4103770 2024-01-23T08:49:25.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.261+0000 7f90fc30d700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f410d790 con 0x7f90f4102dd0 2024-01-23T08:49:25.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.261+0000 7f90f98a8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90f4103770 0x7f90f4113e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:25.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.262+0000 7f90f98a8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90f4103770 0x7f90f4113e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38070/0 (socket says 172.21.15.59:38070) 2024-01-23T08:49:25.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.262+0000 7f90f98a8700 1 -- 172.21.15.59:0/2882397756 learned_addr learned my addr 172.21.15.59:0/2882397756 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:49:25.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.262+0000 7f90fa0a9700 1 --2- 172.21.15.59:0/2882397756 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90f4102dd0 0x7f90f41138d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:25.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.262+0000 7f90f98a8700 1 -- 172.21.15.59:0/2882397756 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90f4102dd0 msgr2=0x7f90f41138d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:25.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.262+0000 7f90f98a8700 1 --2- 172.21.15.59:0/2882397756 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90f4102dd0 0x7f90f41138d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:25.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.262+0000 7f90f98a8700 1 -- 172.21.15.59:0/2882397756 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e4005040 con 0x7f90f4103770 2024-01-23T08:49:25.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.262+0000 7f90f98a8700 1 --2- 172.21.15.59:0/2882397756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90f4103770 0x7f90f4113e10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f90e4009180 tx=0x7f90e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:25.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.262+0000 7f90eaffd700 1 -- 172.21.15.59:0/2882397756 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90e400e040 con 0x7f90f4103770 2024-01-23T08:49:25.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.262+0000 7f90eaffd700 1 -- 172.21.15.59:0/2882397756 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f90e4004290 con 0x7f90f4103770 2024-01-23T08:49:25.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.262+0000 7f90fc30d700 1 -- 172.21.15.59:0/2882397756 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f410da10 con 0x7f90f4103770 2024-01-23T08:49:25.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.262+0000 7f90eaffd700 1 -- 172.21.15.59:0/2882397756 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90e401c3f0 con 0x7f90f4103770 2024-01-23T08:49:25.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.263+0000 7f90fc30d700 1 -- 172.21.15.59:0/2882397756 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f410df50 con 0x7f90f4103770 2024-01-23T08:49:25.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.264+0000 7f90fc30d700 1 -- 172.21.15.59:0/2882397756 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f41052a0 con 0x7f90f4103770 2024-01-23T08:49:25.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.264+0000 7f90eaffd700 1 -- 172.21.15.59:0/2882397756 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f90e401c660 con 0x7f90f4103770 2024-01-23T08:49:25.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.265+0000 7f90eaffd700 1 --2- 172.21.15.59:0/2882397756 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f90e0068f70 0x7f90e006b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:25.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.265+0000 7f90eaffd700 1 -- 172.21.15.59:0/2882397756 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f90e4087bf0 con 0x7f90f4103770 2024-01-23T08:49:25.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.265+0000 7f90fa0a9700 1 --2- 172.21.15.59:0/2882397756 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f90e0068f70 0x7f90e006b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:25.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.266+0000 7f90fa0a9700 1 --2- 172.21.15.59:0/2882397756 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f90e0068f70 0x7f90e006b420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f90f0004f40 tx=0x7f90f000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:25.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.271+0000 7f90eaffd700 1 -- 172.21.15.59:0/2882397756 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f90e4058080 con 0x7f90f4103770 2024-01-23T08:49:25.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.474+0000 7f90fc30d700 1 -- 172.21.15.59:0/2882397756 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f90f41052a0 con 0x7f90f4103770 2024-01-23T08:49:25.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.474+0000 7f90eaffd700 1 -- 172.21.15.59:0/2882397756 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f90e4057c10 con 0x7f90f4103770 2024-01-23T08:49:25.472 INFO:teuthology.orchestra.run.smithi059.stdout:150323855377 2024-01-23T08:49:25.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.476+0000 7f90fc30d700 1 -- 172.21.15.59:0/2882397756 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f90e0068f70 msgr2=0x7f90e006b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:25.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.476+0000 7f90fc30d700 1 --2- 172.21.15.59:0/2882397756 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f90e0068f70 0x7f90e006b420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f90f0004f40 tx=0x7f90f000b040 comp rx=0 tx=0).stop 2024-01-23T08:49:25.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.476+0000 7f90fc30d700 1 -- 172.21.15.59:0/2882397756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90f4103770 msgr2=0x7f90f4113e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:25.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.476+0000 7f90fc30d700 1 --2- 172.21.15.59:0/2882397756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90f4103770 0x7f90f4113e10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f90e4009180 tx=0x7f90e4009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:25.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.476+0000 7f90fc30d700 1 -- 172.21.15.59:0/2882397756 shutdown_connections 2024-01-23T08:49:25.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.476+0000 7f90fc30d700 1 --2- 172.21.15.59:0/2882397756 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f90e0068f70 0x7f90e006b420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:25.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.476+0000 7f90fc30d700 1 --2- 172.21.15.59:0/2882397756 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90f4102dd0 0x7f90f41138d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:25.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.476+0000 7f90fc30d700 1 --2- 172.21.15.59:0/2882397756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90f4103770 0x7f90f4113e10 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:25.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.476+0000 7f90fc30d700 1 -- 172.21.15.59:0/2882397756 >> 172.21.15.59:0/2882397756 conn(0x7f90f40f8980 msgr2=0x7f90f40fa5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:25.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.477+0000 7f90fc30d700 1 -- 172.21.15.59:0/2882397756 shutdown_connections 2024-01-23T08:49:25.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:25.477+0000 7f90fc30d700 1 -- 172.21.15.59:0/2882397756 wait complete. 2024-01-23T08:49:25.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:25 smithi059 ceph-mon[28424]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:25.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:25 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/1217029341' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-01-23T08:49:25.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:25 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:49:25.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:25 smithi137 ceph-mon[35787]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:25.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:25 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/1217029341' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-01-23T08:49:25.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:25 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:49:26.123 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855375 got 150323855377 for osd.4 2024-01-23T08:49:26.123 DEBUG:teuthology.parallel:result is None 2024-01-23T08:49:26.644 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117012 got 115964117014 for osd.3 2024-01-23T08:49:26.645 DEBUG:teuthology.parallel:result is None 2024-01-23T08:49:26.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:26 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/2882397756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-01-23T08:49:26.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:26 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/2882397756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-01-23T08:49:27.080 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705701 got 38654705703 for osd.0 2024-01-23T08:49:27.080 DEBUG:teuthology.parallel:result is None 2024-01-23T08:49:27.080 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-01-23T08:49:27.081 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph pg dump --format=json 2024-01-23T08:49:27.346 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:49:27.687 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:27 smithi059 ceph-mon[28424]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:27.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:27 smithi137 ceph-mon[35787]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:29.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.065+0000 7f18e6f84700 1 -- 172.21.15.59:0/28956611 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18e00fde70 msgr2=0x7f18e00fe240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:29.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.065+0000 7f18e6f84700 1 --2- 172.21.15.59:0/28956611 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18e00fde70 0x7f18e00fe240 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f18d4004970 tx=0x7f18d4009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:29.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.065+0000 7f18e6f84700 1 -- 172.21.15.59:0/28956611 shutdown_connections 2024-01-23T08:49:29.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.065+0000 7f18e6f84700 1 --2- 172.21.15.59:0/28956611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f18e00fe780 0x7f18e0106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:29.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.065+0000 7f18e6f84700 1 --2- 172.21.15.59:0/28956611 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18e00fde70 0x7f18e00fe240 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:29.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.065+0000 7f18e6f84700 1 -- 172.21.15.59:0/28956611 >> 172.21.15.59:0/28956611 conn(0x7f18e0071c20 msgr2=0x7f18e0072020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:29.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.065+0000 7f18e6f84700 1 -- 172.21.15.59:0/28956611 shutdown_connections 2024-01-23T08:49:29.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.065+0000 7f18e6f84700 1 -- 172.21.15.59:0/28956611 wait complete. 2024-01-23T08:49:29.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.066+0000 7f18e6f84700 1 Processor -- start 2024-01-23T08:49:29.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.066+0000 7f18e6f84700 1 -- start start 2024-01-23T08:49:29.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.066+0000 7f18e6f84700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f18e00fde70 0x7f18e0113860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:29.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.066+0000 7f18e6f84700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18e00fe780 0x7f18e0113da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:29.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.066+0000 7f18e6f84700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e0114480 con 0x7f18e00fe780 2024-01-23T08:49:29.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.066+0000 7f18e6f84700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e010d720 con 0x7f18e00fde70 2024-01-23T08:49:29.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.066+0000 7f18dffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18e00fe780 0x7f18e0113da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:29.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.066+0000 7f18dffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18e00fe780 0x7f18e0113da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44396/0 (socket says 172.21.15.59:44396) 2024-01-23T08:49:29.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.066+0000 7f18dffff700 1 -- 172.21.15.59:0/3353214242 learned_addr learned my addr 172.21.15.59:0/3353214242 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:49:29.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.067+0000 7f18e4d20700 1 --2- 172.21.15.59:0/3353214242 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f18e00fde70 0x7f18e0113860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:29.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.067+0000 7f18dffff700 1 -- 172.21.15.59:0/3353214242 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f18e00fde70 msgr2=0x7f18e0113860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:29.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.067+0000 7f18dffff700 1 --2- 172.21.15.59:0/3353214242 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f18e00fde70 0x7f18e0113860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:29.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.067+0000 7f18dffff700 1 -- 172.21.15.59:0/3353214242 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d4005040 con 0x7f18e00fe780 2024-01-23T08:49:29.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.067+0000 7f18dffff700 1 --2- 172.21.15.59:0/3353214242 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18e00fe780 0x7f18e0113da0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f18cc00b790 tx=0x7f18cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:29.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.067+0000 7f18dd7fa700 1 -- 172.21.15.59:0/3353214242 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18cc00f6f0 con 0x7f18e00fe780 2024-01-23T08:49:29.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.067+0000 7f18e6f84700 1 -- 172.21.15.59:0/3353214242 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e010da00 con 0x7f18e00fe780 2024-01-23T08:49:29.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.067+0000 7f18dd7fa700 1 -- 172.21.15.59:0/3353214242 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f18cc00fd00 con 0x7f18e00fe780 2024-01-23T08:49:29.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.067+0000 7f18dd7fa700 1 -- 172.21.15.59:0/3353214242 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18cc0103a0 con 0x7f18e00fe780 2024-01-23T08:49:29.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.067+0000 7f18e6f84700 1 -- 172.21.15.59:0/3353214242 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e010df40 con 0x7f18e00fe780 2024-01-23T08:49:29.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.068+0000 7f18e6f84700 1 -- 172.21.15.59:0/3353214242 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18e0104550 con 0x7f18e00fe780 2024-01-23T08:49:29.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.069+0000 7f18dd7fa700 1 -- 172.21.15.59:0/3353214242 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f18cc00f850 con 0x7f18e00fe780 2024-01-23T08:49:29.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.069+0000 7f18dd7fa700 1 --2- 172.21.15.59:0/3353214242 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f18d0068f60 0x7f18d006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:29.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.069+0000 7f18e4d20700 1 --2- 172.21.15.59:0/3353214242 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f18d0068f60 0x7f18d006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:29.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.069+0000 7f18dd7fa700 1 -- 172.21.15.59:0/3353214242 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f18cc086870 con 0x7f18e00fe780 2024-01-23T08:49:29.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.069+0000 7f18e4d20700 1 --2- 172.21.15.59:0/3353214242 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f18d0068f60 0x7f18d006b410 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f18d4004f50 tx=0x7f18d4009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:29.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.074+0000 7f18dd7fa700 1 -- 172.21.15.59:0/3353214242 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f18cc056b10 con 0x7f18e00fe780 2024-01-23T08:49:29.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.260+0000 7f18e6f84700 1 -- 172.21.15.59:0/3353214242 --> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f18e0109e00 con 0x7f18d0068f60 2024-01-23T08:49:29.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.261+0000 7f18dd7fa700 1 -- 172.21.15.59:0/3353214242 <== mgr.14210 v2:172.21.15.59:6800/3219735162 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28328 (secure 0 0 0) 0x7f18e0109e00 con 0x7f18d0068f60 2024-01-23T08:49:29.260 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:49:29.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.263+0000 7f18e6f84700 1 -- 172.21.15.59:0/3353214242 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f18d0068f60 msgr2=0x7f18d006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:29.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.263+0000 7f18e6f84700 1 --2- 172.21.15.59:0/3353214242 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f18d0068f60 0x7f18d006b410 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f18d4004f50 tx=0x7f18d4009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:29.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.263+0000 7f18e6f84700 1 -- 172.21.15.59:0/3353214242 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18e00fe780 msgr2=0x7f18e0113da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:29.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.263+0000 7f18e6f84700 1 --2- 172.21.15.59:0/3353214242 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18e00fe780 0x7f18e0113da0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f18cc00b790 tx=0x7f18cc009250 comp rx=0 tx=0).stop 2024-01-23T08:49:29.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.264+0000 7f18e6f84700 1 -- 172.21.15.59:0/3353214242 shutdown_connections 2024-01-23T08:49:29.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.264+0000 7f18e6f84700 1 --2- 172.21.15.59:0/3353214242 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f18d0068f60 0x7f18d006b410 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:29.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.264+0000 7f18e6f84700 1 --2- 172.21.15.59:0/3353214242 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f18e00fde70 0x7f18e0113860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:29.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.264+0000 7f18e6f84700 1 --2- 172.21.15.59:0/3353214242 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18e00fe780 0x7f18e0113da0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:29.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.264+0000 7f18e6f84700 1 -- 172.21.15.59:0/3353214242 >> 172.21.15.59:0/3353214242 conn(0x7f18e0071c20 msgr2=0x7f18e01016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:29.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.264+0000 7f18e6f84700 1 -- 172.21.15.59:0/3353214242 shutdown_connections 2024-01-23T08:49:29.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:29.264+0000 7f18e6f84700 1 -- 172.21.15.59:0/3353214242 wait complete. 2024-01-23T08:49:29.263 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-01-23T08:49:29.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:29 smithi059 ceph-mon[28424]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:29.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:29 smithi137 ceph-mon[35787]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:29.980 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-01-23T08:49:28.351055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787652,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683260,"statfs":{"total":575970213888,"available":574139658240,"internally_reserved":0,"allocated":4018176,"data_stored":2624466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002333"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":188,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-01-23T08:48:36.418292+0000","last_change":"2024-01-23T08:48:11.214808+0000","last_active":"2024-01-23T08:48:36.418292+0000","last_peered":"2024-01-23T08:48:36.418292+0000","last_clean":"2024-01-23T08:48:36.418292+0000","last_became_active":"2024-01-23T08:48:11.214490+0000","last_became_peered":"2024-01-23T08:48:11.214490+0000","last_unstale":"2024-01-23T08:48:36.418292+0000","last_undegraded":"2024-01-23T08:48:36.418292+0000","last_fullsized":"2024-01-23T08:48:36.418292+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-23T08:47:25.060382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-23T08:47:25.060382+0000","last_clean_scrub_stamp":"2024-01-23T08:47:25.060382+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-24T19:32:23.037967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 23 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.60799999999999998}]},{"osd":1,"last update":"Tue Jan 23 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.627}]},{"osd":2,"last update":"Tue Jan 23 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.53200000000000003}]},{"osd":3,"last update":"Tue Jan 23 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.45200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117015,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 23 08:48:45 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.44800000000000001}]},{"osd":1,"last update":"Tue Jan 23 08:48:45 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.51000000000000001}]},{"osd":2,"last update":"Tue Jan 23 08:48:45 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.57399999999999995}]},{"osd":4,"last update":"Tue Jan 23 08:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.309},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.36199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 23 08:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.252},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.309}]},{"osd":1,"last update":"Tue Jan 23 08:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.33000000000000002}]},{"osd":3,"last update":"Tue Jan 23 08:48:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.48199999999999998}]},{"osd":4,"last update":"Tue Jan 23 08:49:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.495}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 23 08:49:00 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.33600000000000002,"15min":0.33300000000000002},"min":{"1min":0.246,"5min":0.221,"15min":0.221},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.438},{"interface":"front","average":{"1min":0.377,"5min":0.36399999999999999,"15min":0.36199999999999999},"min":{"1min":0.23699999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.33200000000000002}]},{"osd":2,"last update":"Tue Jan 23 08:48:28 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.374}]},{"osd":3,"last update":"Tue Jan 23 08:48:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.54200000000000004}]},{"osd":4,"last update":"Tue Jan 23 08:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.52000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 23 08:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.32700000000000001,"15min":0.32500000000000001},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.45900000000000002,"5min":0.47899999999999998,"15min":0.47899999999999998},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.34100000000000003,"15min":0.33800000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.317}]},{"osd":2,"last update":"Tue Jan 23 08:48:25 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.28199999999999997}]},{"osd":3,"last update":"Tue Jan 23 08:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.59199999999999997}]},{"osd":4,"last update":"Tue Jan 23 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.53400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-23T08:49:29.981 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph pg dump --format=json 2024-01-23T08:49:30.250 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:49:30.703 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:30 smithi059 ceph-mon[28424]: from='client.14422 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-23T08:49:30.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:30 smithi137 ceph-mon[35787]: from='client.14422 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-23T08:49:31.663 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:31 smithi059 ceph-mon[28424]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:31.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:31 smithi137 ceph-mon[35787]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.863+0000 7fbf6ac56700 1 -- 172.21.15.59:0/125733748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf64101cf0 msgr2=0x7fbf641020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.863+0000 7fbf6ac56700 1 --2- 172.21.15.59:0/125733748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf64101cf0 0x7fbf641020c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c009ce0 tx=0x7fbf5c004d10 comp rx=0 tx=0).stop 2024-01-23T08:49:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.863+0000 7fbf6ac56700 1 -- 172.21.15.59:0/125733748 shutdown_connections 2024-01-23T08:49:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.863+0000 7fbf6ac56700 1 --2- 172.21.15.59:0/125733748 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbf64102690 0x7fbf640fbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.863+0000 7fbf6ac56700 1 --2- 172.21.15.59:0/125733748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf64101cf0 0x7fbf641020c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.863+0000 7fbf6ac56700 1 -- 172.21.15.59:0/125733748 >> 172.21.15.59:0/125733748 conn(0x7fbf640f78a0 msgr2=0x7fbf640f9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.863+0000 7fbf6ac56700 1 -- 172.21.15.59:0/125733748 shutdown_connections 2024-01-23T08:49:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.863+0000 7fbf6ac56700 1 -- 172.21.15.59:0/125733748 wait complete. 2024-01-23T08:49:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.864+0000 7fbf6ac56700 1 Processor -- start 2024-01-23T08:49:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.864+0000 7fbf6ac56700 1 -- start start 2024-01-23T08:49:31.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.864+0000 7fbf6ac56700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbf64101cf0 0x7fbf64112910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:31.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.864+0000 7fbf6ac56700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf64102690 0x7fbf64112e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:31.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.864+0000 7fbf6ac56700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf64113530 con 0x7fbf64102690 2024-01-23T08:49:31.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.864+0000 7fbf6ac56700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6410c7d0 con 0x7fbf64101cf0 2024-01-23T08:49:31.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.864+0000 7fbf63fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf64102690 0x7fbf64112e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:31.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.864+0000 7fbf63fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf64102690 0x7fbf64112e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44404/0 (socket says 172.21.15.59:44404) 2024-01-23T08:49:31.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.864+0000 7fbf63fff700 1 -- 172.21.15.59:0/2952434464 learned_addr learned my addr 172.21.15.59:0/2952434464 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:49:31.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.865+0000 7fbf689f2700 1 --2- 172.21.15.59:0/2952434464 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbf64101cf0 0x7fbf64112910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:31.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.865+0000 7fbf63fff700 1 -- 172.21.15.59:0/2952434464 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbf64101cf0 msgr2=0x7fbf64112910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:31.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.865+0000 7fbf63fff700 1 --2- 172.21.15.59:0/2952434464 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbf64101cf0 0x7fbf64112910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:31.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.865+0000 7fbf63fff700 1 -- 172.21.15.59:0/2952434464 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c009940 con 0x7fbf64102690 2024-01-23T08:49:31.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.865+0000 7fbf63fff700 1 --2- 172.21.15.59:0/2952434464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf64102690 0x7fbf64112e50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fbf5800b790 tx=0x7fbf58009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:31.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.865+0000 7fbf617fa700 1 -- 172.21.15.59:0/2952434464 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf5800f6f0 con 0x7fbf64102690 2024-01-23T08:49:31.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.865+0000 7fbf6ac56700 1 -- 172.21.15.59:0/2952434464 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf6410cab0 con 0x7fbf64102690 2024-01-23T08:49:31.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.865+0000 7fbf617fa700 1 -- 172.21.15.59:0/2952434464 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf5800fd00 con 0x7fbf64102690 2024-01-23T08:49:31.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.865+0000 7fbf617fa700 1 -- 172.21.15.59:0/2952434464 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf580103a0 con 0x7fbf64102690 2024-01-23T08:49:31.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.865+0000 7fbf6ac56700 1 -- 172.21.15.59:0/2952434464 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf6410cff0 con 0x7fbf64102690 2024-01-23T08:49:31.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.866+0000 7fbf6ac56700 1 -- 172.21.15.59:0/2952434464 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf640ff000 con 0x7fbf64102690 2024-01-23T08:49:31.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.867+0000 7fbf617fa700 1 -- 172.21.15.59:0/2952434464 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fbf5800f850 con 0x7fbf64102690 2024-01-23T08:49:31.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.867+0000 7fbf617fa700 1 --2- 172.21.15.59:0/2952434464 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbf4c068f60 0x7fbf4c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:31.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.867+0000 7fbf617fa700 1 -- 172.21.15.59:0/2952434464 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbf58086810 con 0x7fbf64102690 2024-01-23T08:49:31.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.867+0000 7fbf689f2700 1 --2- 172.21.15.59:0/2952434464 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbf4c068f60 0x7fbf4c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:31.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.867+0000 7fbf689f2700 1 --2- 172.21.15.59:0/2952434464 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbf4c068f60 0x7fbf4c06b410 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c015040 tx=0x7fbf5c004d10 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:31.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:31.871+0000 7fbf617fa700 1 -- 172.21.15.59:0/2952434464 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fbf58056ab0 con 0x7fbf64102690 2024-01-23T08:49:32.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:32.057+0000 7fbf6ac56700 1 -- 172.21.15.59:0/2952434464 --> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbf64108d20 con 0x7fbf4c068f60 2024-01-23T08:49:32.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:32.058+0000 7fbf617fa700 1 -- 172.21.15.59:0/2952434464 <== mgr.14210 v2:172.21.15.59:6800/3219735162 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28328 (secure 0 0 0) 0x7fbf64108d20 con 0x7fbf4c068f60 2024-01-23T08:49:32.057 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:49:32.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:32.060+0000 7fbf6ac56700 1 -- 172.21.15.59:0/2952434464 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbf4c068f60 msgr2=0x7fbf4c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:32.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:32.060+0000 7fbf6ac56700 1 --2- 172.21.15.59:0/2952434464 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbf4c068f60 0x7fbf4c06b410 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c015040 tx=0x7fbf5c004d10 comp rx=0 tx=0).stop 2024-01-23T08:49:32.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:32.060+0000 7fbf6ac56700 1 -- 172.21.15.59:0/2952434464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf64102690 msgr2=0x7fbf64112e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:32.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:32.060+0000 7fbf6ac56700 1 --2- 172.21.15.59:0/2952434464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf64102690 0x7fbf64112e50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fbf5800b790 tx=0x7fbf58009220 comp rx=0 tx=0).stop 2024-01-23T08:49:32.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:32.060+0000 7fbf6ac56700 1 -- 172.21.15.59:0/2952434464 shutdown_connections 2024-01-23T08:49:32.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:32.060+0000 7fbf6ac56700 1 --2- 172.21.15.59:0/2952434464 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbf4c068f60 0x7fbf4c06b410 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:32.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:32.060+0000 7fbf6ac56700 1 --2- 172.21.15.59:0/2952434464 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbf64101cf0 0x7fbf64112910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:32.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:32.060+0000 7fbf6ac56700 1 --2- 172.21.15.59:0/2952434464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf64102690 0x7fbf64112e50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:32.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:32.060+0000 7fbf6ac56700 1 -- 172.21.15.59:0/2952434464 >> 172.21.15.59:0/2952434464 conn(0x7fbf640f78a0 msgr2=0x7fbf640f9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:32.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:32.061+0000 7fbf6ac56700 1 -- 172.21.15.59:0/2952434464 shutdown_connections 2024-01-23T08:49:32.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:32.061+0000 7fbf6ac56700 1 -- 172.21.15.59:0/2952434464 wait complete. 2024-01-23T08:49:32.061 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-01-23T08:49:32.816 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-01-23T08:49:30.351416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787652,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683260,"statfs":{"total":575970213888,"available":574139658240,"internally_reserved":0,"allocated":4018176,"data_stored":2624466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002349"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":188,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-01-23T08:48:36.418292+0000","last_change":"2024-01-23T08:48:11.214808+0000","last_active":"2024-01-23T08:48:36.418292+0000","last_peered":"2024-01-23T08:48:36.418292+0000","last_clean":"2024-01-23T08:48:36.418292+0000","last_became_active":"2024-01-23T08:48:11.214490+0000","last_became_peered":"2024-01-23T08:48:11.214490+0000","last_unstale":"2024-01-23T08:48:36.418292+0000","last_undegraded":"2024-01-23T08:48:36.418292+0000","last_fullsized":"2024-01-23T08:48:36.418292+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-23T08:47:25.060382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-23T08:47:25.060382+0000","last_clean_scrub_stamp":"2024-01-23T08:47:25.060382+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-24T19:32:23.037967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 23 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.60799999999999998}]},{"osd":1,"last update":"Tue Jan 23 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.627}]},{"osd":2,"last update":"Tue Jan 23 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.53200000000000003}]},{"osd":3,"last update":"Tue Jan 23 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.45200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117015,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 23 08:48:45 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.44800000000000001}]},{"osd":1,"last update":"Tue Jan 23 08:48:45 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.51000000000000001}]},{"osd":2,"last update":"Tue Jan 23 08:48:45 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.57399999999999995}]},{"osd":4,"last update":"Tue Jan 23 08:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.309},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.36199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 23 08:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.252},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.309}]},{"osd":1,"last update":"Tue Jan 23 08:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.33000000000000002}]},{"osd":3,"last update":"Tue Jan 23 08:48:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.48199999999999998}]},{"osd":4,"last update":"Tue Jan 23 08:49:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.495}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 23 08:49:00 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.33600000000000002,"15min":0.33300000000000002},"min":{"1min":0.246,"5min":0.221,"15min":0.221},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.438},{"interface":"front","average":{"1min":0.377,"5min":0.36399999999999999,"15min":0.36199999999999999},"min":{"1min":0.23699999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.33200000000000002}]},{"osd":2,"last update":"Tue Jan 23 08:48:28 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.374}]},{"osd":3,"last update":"Tue Jan 23 08:48:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.54200000000000004}]},{"osd":4,"last update":"Tue Jan 23 08:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.52000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 23 08:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.32700000000000001,"15min":0.32500000000000001},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.45900000000000002,"5min":0.47899999999999998,"15min":0.47899999999999998},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.34100000000000003,"15min":0.33800000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.317}]},{"osd":2,"last update":"Tue Jan 23 08:48:25 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.28199999999999997}]},{"osd":3,"last update":"Tue Jan 23 08:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.59199999999999997}]},{"osd":4,"last update":"Tue Jan 23 08:49:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.53400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-23T08:49:32.817 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-01-23T08:49:32.818 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-23T08:49:32.818 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-01-23T08:49:32.818 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph health --format=json 2024-01-23T08:49:33.078 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:49:33.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:33 smithi059 ceph-mon[28424]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-23T08:49:33.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:33 smithi059 ceph-mon[28424]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:33.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:33 smithi137 ceph-mon[35787]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-23T08:49:33.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:33 smithi137 ceph-mon[35787]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:34.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.682+0000 7fc9cf311700 1 -- 172.21.15.59:0/259515973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c8103770 msgr2=0x7fc9c80fcde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:34.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.682+0000 7fc9cf311700 1 --2- 172.21.15.59:0/259515973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c8103770 0x7fc9c80fcde0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc004940 tx=0x7fc9bc009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:34.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.682+0000 7fc9cf311700 1 -- 172.21.15.59:0/259515973 shutdown_connections 2024-01-23T08:49:34.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.682+0000 7fc9cf311700 1 --2- 172.21.15.59:0/259515973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c8103770 0x7fc9c80fcde0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:34.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.682+0000 7fc9cf311700 1 --2- 172.21.15.59:0/259515973 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9c8102dd0 0x7fc9c81031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:34.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.682+0000 7fc9cf311700 1 -- 172.21.15.59:0/259515973 >> 172.21.15.59:0/259515973 conn(0x7fc9c80f8980 msgr2=0x7fc9c80fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:34.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.682+0000 7fc9cf311700 1 -- 172.21.15.59:0/259515973 shutdown_connections 2024-01-23T08:49:34.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.682+0000 7fc9cf311700 1 -- 172.21.15.59:0/259515973 wait complete. 2024-01-23T08:49:34.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.683+0000 7fc9cf311700 1 Processor -- start 2024-01-23T08:49:34.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.683+0000 7fc9cf311700 1 -- start start 2024-01-23T08:49:34.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.683+0000 7fc9cf311700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9c8102dd0 0x7fc9c81138c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:34.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.683+0000 7fc9cf311700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c8103770 0x7fc9c8113e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:34.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.683+0000 7fc9cf311700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c81144e0 con 0x7fc9c8103770 2024-01-23T08:49:34.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.683+0000 7fc9cf311700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c810d780 con 0x7fc9c8102dd0 2024-01-23T08:49:34.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.683+0000 7fc9cc8ac700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c8103770 0x7fc9c8113e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:34.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.683+0000 7fc9cc8ac700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c8103770 0x7fc9c8113e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44410/0 (socket says 172.21.15.59:44410) 2024-01-23T08:49:34.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.683+0000 7fc9cc8ac700 1 -- 172.21.15.59:0/3149872885 learned_addr learned my addr 172.21.15.59:0/3149872885 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:49:34.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.684+0000 7fc9cd0ad700 1 --2- 172.21.15.59:0/3149872885 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9c8102dd0 0x7fc9c81138c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:34.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.684+0000 7fc9cc8ac700 1 -- 172.21.15.59:0/3149872885 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9c8102dd0 msgr2=0x7fc9c81138c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:34.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.684+0000 7fc9cc8ac700 1 --2- 172.21.15.59:0/3149872885 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9c8102dd0 0x7fc9c81138c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:34.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.684+0000 7fc9cc8ac700 1 -- 172.21.15.59:0/3149872885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9bc005040 con 0x7fc9c8103770 2024-01-23T08:49:34.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.684+0000 7fc9cd0ad700 1 --2- 172.21.15.59:0/3149872885 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9c8102dd0 0x7fc9c81138c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-23T08:49:34.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.684+0000 7fc9cc8ac700 1 --2- 172.21.15.59:0/3149872885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c8103770 0x7fc9c8113e00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc009180 tx=0x7fc9bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:34.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.684+0000 7fc9c5ffb700 1 -- 172.21.15.59:0/3149872885 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9bc00e040 con 0x7fc9c8103770 2024-01-23T08:49:34.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.684+0000 7fc9cf311700 1 -- 172.21.15.59:0/3149872885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c810da00 con 0x7fc9c8103770 2024-01-23T08:49:34.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.684+0000 7fc9c5ffb700 1 -- 172.21.15.59:0/3149872885 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc9bc004290 con 0x7fc9c8103770 2024-01-23T08:49:34.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.684+0000 7fc9c5ffb700 1 -- 172.21.15.59:0/3149872885 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9bc01c3f0 con 0x7fc9c8103770 2024-01-23T08:49:34.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.684+0000 7fc9cf311700 1 -- 172.21.15.59:0/3149872885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c810df40 con 0x7fc9c8103770 2024-01-23T08:49:34.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.685+0000 7fc9cf311700 1 -- 172.21.15.59:0/3149872885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c81000e0 con 0x7fc9c8103770 2024-01-23T08:49:34.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.686+0000 7fc9c5ffb700 1 -- 172.21.15.59:0/3149872885 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fc9bc00bc50 con 0x7fc9c8103770 2024-01-23T08:49:34.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.686+0000 7fc9c5ffb700 1 --2- 172.21.15.59:0/3149872885 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fc9b0068f70 0x7fc9b006b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:34.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.686+0000 7fc9c5ffb700 1 -- 172.21.15.59:0/3149872885 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc9bc087dc0 con 0x7fc9c8103770 2024-01-23T08:49:34.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.686+0000 7fc9cd0ad700 1 --2- 172.21.15.59:0/3149872885 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fc9b0068f70 0x7fc9b006b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:34.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.686+0000 7fc9cd0ad700 1 --2- 172.21.15.59:0/3149872885 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fc9b0068f70 0x7fc9b006b420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc9c0007fd0 tx=0x7fc9c00074f0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:34.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.690+0000 7fc9c5ffb700 1 -- 172.21.15.59:0/3149872885 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc9bc058110 con 0x7fc9c8103770 2024-01-23T08:49:34.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.922+0000 7fc9cf311700 1 -- 172.21.15.59:0/3149872885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc9c80fc580 con 0x7fc9c8103770 2024-01-23T08:49:34.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.922+0000 7fc9c5ffb700 1 -- 172.21.15.59:0/3149872885 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fc9bc057ca0 con 0x7fc9c8103770 2024-01-23T08:49:34.921 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:49:34.921 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-23T08:49:34.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.924+0000 7fc9cf311700 1 -- 172.21.15.59:0/3149872885 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fc9b0068f70 msgr2=0x7fc9b006b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:34.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.924+0000 7fc9cf311700 1 --2- 172.21.15.59:0/3149872885 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fc9b0068f70 0x7fc9b006b420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc9c0007fd0 tx=0x7fc9c00074f0 comp rx=0 tx=0).stop 2024-01-23T08:49:34.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.924+0000 7fc9cf311700 1 -- 172.21.15.59:0/3149872885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c8103770 msgr2=0x7fc9c8113e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:34.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.924+0000 7fc9cf311700 1 --2- 172.21.15.59:0/3149872885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c8103770 0x7fc9c8113e00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc009180 tx=0x7fc9bc009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:34.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.924+0000 7fc9cf311700 1 -- 172.21.15.59:0/3149872885 shutdown_connections 2024-01-23T08:49:34.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.924+0000 7fc9cf311700 1 --2- 172.21.15.59:0/3149872885 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fc9b0068f70 0x7fc9b006b420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:34.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.924+0000 7fc9cf311700 1 --2- 172.21.15.59:0/3149872885 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9c8102dd0 0x7fc9c81138c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:34.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.924+0000 7fc9cf311700 1 --2- 172.21.15.59:0/3149872885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9c8103770 0x7fc9c8113e00 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:34.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.924+0000 7fc9cf311700 1 -- 172.21.15.59:0/3149872885 >> 172.21.15.59:0/3149872885 conn(0x7fc9c80f8980 msgr2=0x7fc9c80fa3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:34.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.924+0000 7fc9cf311700 1 -- 172.21.15.59:0/3149872885 shutdown_connections 2024-01-23T08:49:34.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:34.924+0000 7fc9cf311700 1 -- 172.21.15.59:0/3149872885 wait complete. 2024-01-23T08:49:35.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:35 smithi059 ceph-mon[28424]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:35.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:35 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/3149872885' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-23T08:49:35.612 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-01-23T08:49:35.612 INFO:tasks.cephadm:Setup complete, yielding 2024-01-23T08:49:35.613 INFO:teuthology.run_tasks:Running task print... 2024-01-23T08:49:35.626 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-01-23T08:49:35.626 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-01-23T08:49:35.667 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2024-01-23T08:49:35.668 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-01-23T08:49:35.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:35 smithi137 ceph-mon[35787]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:35.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:35 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/3149872885' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-23T08:49:35.927 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:49:37.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.531+0000 7f4c5df0c700 1 -- 172.21.15.59:0/623837674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c580fdfe0 msgr2=0x7f4c580fe430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:37.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.531+0000 7f4c5df0c700 1 --2- 172.21.15.59:0/623837674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c580fdfe0 0x7f4c580fe430 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c004940 tx=0x7f4c4c009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:37.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.531+0000 7f4c5df0c700 1 -- 172.21.15.59:0/623837674 shutdown_connections 2024-01-23T08:49:37.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.531+0000 7f4c5df0c700 1 --2- 172.21.15.59:0/623837674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c580fdfe0 0x7f4c580fe430 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:37.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.531+0000 7f4c5df0c700 1 --2- 172.21.15.59:0/623837674 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c580fcdd0 0x7f4c580fd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:37.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.531+0000 7f4c5df0c700 1 -- 172.21.15.59:0/623837674 >> 172.21.15.59:0/623837674 conn(0x7f4c580f84b0 msgr2=0x7f4c580fa8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:37.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.531+0000 7f4c5df0c700 1 -- 172.21.15.59:0/623837674 shutdown_connections 2024-01-23T08:49:37.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.531+0000 7f4c5df0c700 1 -- 172.21.15.59:0/623837674 wait complete. 2024-01-23T08:49:37.531 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:37 smithi059 ceph-mon[28424]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:37.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.532+0000 7f4c5df0c700 1 Processor -- start 2024-01-23T08:49:37.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.532+0000 7f4c5df0c700 1 -- start start 2024-01-23T08:49:37.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.532+0000 7f4c5df0c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c580fcdd0 0x7f4c5810d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:37.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.532+0000 7f4c5df0c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c580fdfe0 0x7f4c5810dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:37.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.532+0000 7f4c5df0c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5810e310 con 0x7f4c580fdfe0 2024-01-23T08:49:37.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.532+0000 7f4c5df0c700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5810eb50 con 0x7f4c580fcdd0 2024-01-23T08:49:37.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.532+0000 7f4c56ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c580fdfe0 0x7f4c5810dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:37.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.532+0000 7f4c56ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c580fdfe0 0x7f4c5810dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34160/0 (socket says 172.21.15.59:34160) 2024-01-23T08:49:37.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.532+0000 7f4c56ffd700 1 -- 172.21.15.59:0/2419222304 learned_addr learned my addr 172.21.15.59:0/2419222304 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:49:37.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.533+0000 7f4c577fe700 1 --2- 172.21.15.59:0/2419222304 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c580fcdd0 0x7f4c5810d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:37.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.533+0000 7f4c56ffd700 1 -- 172.21.15.59:0/2419222304 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c580fcdd0 msgr2=0x7f4c5810d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:37.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.533+0000 7f4c56ffd700 1 --2- 172.21.15.59:0/2419222304 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c580fcdd0 0x7f4c5810d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:37.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.533+0000 7f4c56ffd700 1 -- 172.21.15.59:0/2419222304 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4c005040 con 0x7f4c580fdfe0 2024-01-23T08:49:37.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.533+0000 7f4c56ffd700 1 --2- 172.21.15.59:0/2419222304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c580fdfe0 0x7f4c5810dc80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009380 tx=0x7f4c4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:37.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.533+0000 7f4c3ffff700 1 -- 172.21.15.59:0/2419222304 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c4c00e040 con 0x7f4c580fdfe0 2024-01-23T08:49:37.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.533+0000 7f4c3ffff700 1 -- 172.21.15.59:0/2419222304 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c4c004290 con 0x7f4c580fdfe0 2024-01-23T08:49:37.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.533+0000 7f4c3ffff700 1 -- 172.21.15.59:0/2419222304 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c4c013ea0 con 0x7f4c580fdfe0 2024-01-23T08:49:37.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.533+0000 7f4c5df0c700 1 -- 172.21.15.59:0/2419222304 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c5810edd0 con 0x7f4c580fdfe0 2024-01-23T08:49:37.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.533+0000 7f4c5df0c700 1 -- 172.21.15.59:0/2419222304 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c5810f2c0 con 0x7f4c580fdfe0 2024-01-23T08:49:37.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.534+0000 7f4c5df0c700 1 -- 172.21.15.59:0/2419222304 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c5804bd60 con 0x7f4c580fdfe0 2024-01-23T08:49:37.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.535+0000 7f4c3ffff700 1 -- 172.21.15.59:0/2419222304 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f4c4c00bc50 con 0x7f4c580fdfe0 2024-01-23T08:49:37.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.535+0000 7f4c3ffff700 1 --2- 172.21.15.59:0/2419222304 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f4c40068f70 0x7f4c4006b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:37.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.535+0000 7f4c3ffff700 1 -- 172.21.15.59:0/2419222304 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4c4c087e90 con 0x7f4c580fdfe0 2024-01-23T08:49:37.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.535+0000 7f4c577fe700 1 --2- 172.21.15.59:0/2419222304 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f4c40068f70 0x7f4c4006b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:37.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.536+0000 7f4c577fe700 1 --2- 172.21.15.59:0/2419222304 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f4c40068f70 0x7f4c4006b420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4c580fde40 tx=0x7f4c480091a0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:37.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.540+0000 7f4c3ffff700 1 -- 172.21.15.59:0/2419222304 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f4c4c0581e0 con 0x7f4c580fdfe0 2024-01-23T08:49:37.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.730+0000 7f4c5df0c700 1 -- 172.21.15.59:0/2419222304 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f4c58105810 con 0x7f4c580fdfe0 2024-01-23T08:49:37.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.736+0000 7f4c3ffff700 1 -- 172.21.15.59:0/2419222304 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f4c4c057d70 con 0x7f4c580fdfe0 2024-01-23T08:49:37.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.739+0000 7f4c5df0c700 1 -- 172.21.15.59:0/2419222304 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f4c40068f70 msgr2=0x7f4c4006b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:37.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.739+0000 7f4c5df0c700 1 --2- 172.21.15.59:0/2419222304 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f4c40068f70 0x7f4c4006b420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4c580fde40 tx=0x7f4c480091a0 comp rx=0 tx=0).stop 2024-01-23T08:49:37.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.739+0000 7f4c5df0c700 1 -- 172.21.15.59:0/2419222304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c580fdfe0 msgr2=0x7f4c5810dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:37.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.739+0000 7f4c5df0c700 1 --2- 172.21.15.59:0/2419222304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c580fdfe0 0x7f4c5810dc80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009380 tx=0x7f4c4c009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:37.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.739+0000 7f4c5df0c700 1 -- 172.21.15.59:0/2419222304 shutdown_connections 2024-01-23T08:49:37.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.739+0000 7f4c5df0c700 1 --2- 172.21.15.59:0/2419222304 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f4c40068f70 0x7f4c4006b420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:37.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.739+0000 7f4c5df0c700 1 --2- 172.21.15.59:0/2419222304 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c580fcdd0 0x7f4c5810d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:37.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.739+0000 7f4c5df0c700 1 --2- 172.21.15.59:0/2419222304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c580fdfe0 0x7f4c5810dc80 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:37.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.739+0000 7f4c5df0c700 1 -- 172.21.15.59:0/2419222304 >> 172.21.15.59:0/2419222304 conn(0x7f4c580f84b0 msgr2=0x7f4c58101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:37.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.739+0000 7f4c5df0c700 1 -- 172.21.15.59:0/2419222304 shutdown_connections 2024-01-23T08:49:37.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:37.739+0000 7f4c5df0c700 1 -- 172.21.15.59:0/2419222304 wait complete. 2024-01-23T08:49:37.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:37 smithi137 ceph-mon[35787]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:38.408 INFO:teuthology.run_tasks:Running task print... 2024-01-23T08:49:38.418 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-01-23T08:49:38.418 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-01-23T08:49:38.441 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2024-01-23T08:49:38.441 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- bash -c 'ceph orch status' 2024-01-23T08:49:38.699 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:49:39.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:38 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/2419222304' entity='client.admin' 2024-01-23T08:49:39.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:38 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:49:39.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:38 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:49:39.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:38 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:49:39.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:38 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:49:39.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:38 smithi137 ceph-mon[35787]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:39.071 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:38 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/2419222304' entity='client.admin' 2024-01-23T08:49:39.071 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:38 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:49:39.071 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:38 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:49:39.071 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:38 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:49:39.072 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:38 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:49:39.072 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:38 smithi059 ceph-mon[28424]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:40.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.325+0000 7eff1f730700 1 -- 172.21.15.59:0/2763588856 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff18066b00 msgr2=0x7eff180ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:40.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.325+0000 7eff1f730700 1 --2- 172.21.15.59:0/2763588856 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff18066b00 0x7eff180ff980 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7eff0c004940 tx=0x7eff0c009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:40.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.325+0000 7eff1f730700 1 -- 172.21.15.59:0/2763588856 shutdown_connections 2024-01-23T08:49:40.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.325+0000 7eff1f730700 1 --2- 172.21.15.59:0/2763588856 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff180fff50 0x7eff181023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:40.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.325+0000 7eff1f730700 1 --2- 172.21.15.59:0/2763588856 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff18066b00 0x7eff180ff980 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:40.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.325+0000 7eff1f730700 1 -- 172.21.15.59:0/2763588856 >> 172.21.15.59:0/2763588856 conn(0x7eff180f73b0 msgr2=0x7eff180f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:40.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.325+0000 7eff1f730700 1 -- 172.21.15.59:0/2763588856 shutdown_connections 2024-01-23T08:49:40.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.325+0000 7eff1f730700 1 -- 172.21.15.59:0/2763588856 wait complete. 2024-01-23T08:49:40.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.326+0000 7eff1f730700 1 Processor -- start 2024-01-23T08:49:40.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.326+0000 7eff1f730700 1 -- start start 2024-01-23T08:49:40.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.326+0000 7eff1f730700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff18066b00 0x7eff180fd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:40.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.326+0000 7eff1f730700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff180fff50 0x7eff180fb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:40.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.326+0000 7eff1f730700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff180fbf10 con 0x7eff180fff50 2024-01-23T08:49:40.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.326+0000 7eff1f730700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff180fc080 con 0x7eff18066b00 2024-01-23T08:49:40.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.326+0000 7eff1cccb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff180fff50 0x7eff180fb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:40.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.326+0000 7eff1cccb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff180fff50 0x7eff180fb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34192/0 (socket says 172.21.15.59:34192) 2024-01-23T08:49:40.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.326+0000 7eff1cccb700 1 -- 172.21.15.59:0/4048477885 learned_addr learned my addr 172.21.15.59:0/4048477885 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:49:40.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.327+0000 7eff1d4cc700 1 --2- 172.21.15.59:0/4048477885 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff18066b00 0x7eff180fd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:40.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.327+0000 7eff1cccb700 1 -- 172.21.15.59:0/4048477885 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff18066b00 msgr2=0x7eff180fd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:40.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.327+0000 7eff1cccb700 1 --2- 172.21.15.59:0/4048477885 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff18066b00 0x7eff180fd400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:40.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.327+0000 7eff1cccb700 1 -- 172.21.15.59:0/4048477885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff0c005040 con 0x7eff180fff50 2024-01-23T08:49:40.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.327+0000 7eff1cccb700 1 --2- 172.21.15.59:0/4048477885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff180fff50 0x7eff180fb880 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7eff1400b790 tx=0x7eff14009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:40.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.327+0000 7eff09ffb700 1 -- 172.21.15.59:0/4048477885 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff1400f6f0 con 0x7eff180fff50 2024-01-23T08:49:40.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.327+0000 7eff09ffb700 1 -- 172.21.15.59:0/4048477885 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff1400fd00 con 0x7eff180fff50 2024-01-23T08:49:40.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.327+0000 7eff1f730700 1 -- 172.21.15.59:0/4048477885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff180fc360 con 0x7eff180fff50 2024-01-23T08:49:40.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.327+0000 7eff09ffb700 1 -- 172.21.15.59:0/4048477885 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff140103a0 con 0x7eff180fff50 2024-01-23T08:49:40.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.327+0000 7eff1f730700 1 -- 172.21.15.59:0/4048477885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff180fc6e0 con 0x7eff180fff50 2024-01-23T08:49:40.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.328+0000 7eff1f730700 1 -- 172.21.15.59:0/4048477885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff1804bd60 con 0x7eff180fff50 2024-01-23T08:49:40.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.329+0000 7eff09ffb700 1 -- 172.21.15.59:0/4048477885 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7eff1400f850 con 0x7eff180fff50 2024-01-23T08:49:40.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.329+0000 7eff09ffb700 1 --2- 172.21.15.59:0/4048477885 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7eff04068f60 0x7eff0406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:40.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.329+0000 7eff09ffb700 1 -- 172.21.15.59:0/4048477885 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7eff140868d0 con 0x7eff180fff50 2024-01-23T08:49:40.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.329+0000 7eff1d4cc700 1 --2- 172.21.15.59:0/4048477885 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7eff04068f60 0x7eff0406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:40.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.330+0000 7eff1d4cc700 1 --2- 172.21.15.59:0/4048477885 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7eff04068f60 0x7eff0406b410 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7eff0c00b3b0 tx=0x7eff0c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:40.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.334+0000 7eff09ffb700 1 -- 172.21.15.59:0/4048477885 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7eff14056af0 con 0x7eff180fff50 2024-01-23T08:49:40.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.558+0000 7eff1f730700 1 -- 172.21.15.59:0/4048477885 --> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7eff18100c30 con 0x7eff04068f60 2024-01-23T08:49:40.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.559+0000 7eff09ffb700 1 -- 172.21.15.59:0/4048477885 <== mgr.14210 v2:172.21.15.59:6800/3219735162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7eff18100c30 con 0x7eff04068f60 2024-01-23T08:49:40.557 INFO:teuthology.orchestra.run.smithi059.stdout:Backend: cephadm 2024-01-23T08:49:40.557 INFO:teuthology.orchestra.run.smithi059.stdout:Available: Yes 2024-01-23T08:49:40.557 INFO:teuthology.orchestra.run.smithi059.stdout:Paused: No 2024-01-23T08:49:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.561+0000 7eff1f730700 1 -- 172.21.15.59:0/4048477885 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7eff04068f60 msgr2=0x7eff0406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.561+0000 7eff1f730700 1 --2- 172.21.15.59:0/4048477885 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7eff04068f60 0x7eff0406b410 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7eff0c00b3b0 tx=0x7eff0c009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.561+0000 7eff1f730700 1 -- 172.21.15.59:0/4048477885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff180fff50 msgr2=0x7eff180fb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.561+0000 7eff1f730700 1 --2- 172.21.15.59:0/4048477885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff180fff50 0x7eff180fb880 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7eff1400b790 tx=0x7eff14009220 comp rx=0 tx=0).stop 2024-01-23T08:49:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.561+0000 7eff1f730700 1 -- 172.21.15.59:0/4048477885 shutdown_connections 2024-01-23T08:49:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.561+0000 7eff1f730700 1 --2- 172.21.15.59:0/4048477885 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7eff04068f60 0x7eff0406b410 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.561+0000 7eff1f730700 1 --2- 172.21.15.59:0/4048477885 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff18066b00 0x7eff180fd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.561+0000 7eff1f730700 1 --2- 172.21.15.59:0/4048477885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff180fff50 0x7eff180fb880 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.561+0000 7eff1f730700 1 -- 172.21.15.59:0/4048477885 >> 172.21.15.59:0/4048477885 conn(0x7eff180f73b0 msgr2=0x7eff180f9790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.561+0000 7eff1f730700 1 -- 172.21.15.59:0/4048477885 shutdown_connections 2024-01-23T08:49:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:40.561+0000 7eff1f730700 1 -- 172.21.15.59:0/4048477885 wait complete. 2024-01-23T08:49:41.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:41 smithi059 ceph-mon[28424]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:41.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:41 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:49:41.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:41 smithi059 ceph-mon[28424]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:49:41.288 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- bash -c 'ceph orch ps' 2024-01-23T08:49:41.292 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:41 smithi137 ceph-mon[35787]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:41.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:41 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:49:41.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:41 smithi137 ceph-mon[35787]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:49:41.559 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:49:43.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.218+0000 7f0547aec700 1 -- 172.21.15.59:0/280512986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05400fdfe0 msgr2=0x7f05400fe430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:43.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.218+0000 7f0547aec700 1 --2- 172.21.15.59:0/280512986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05400fdfe0 0x7f05400fe430 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f053c004940 tx=0x7f053c009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:43.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.218+0000 7f0547aec700 1 -- 172.21.15.59:0/280512986 shutdown_connections 2024-01-23T08:49:43.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.218+0000 7f0547aec700 1 --2- 172.21.15.59:0/280512986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05400fdfe0 0x7f05400fe430 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:43.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.218+0000 7f0547aec700 1 --2- 172.21.15.59:0/280512986 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05400fcdd0 0x7f05400fd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:43.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.218+0000 7f0547aec700 1 -- 172.21.15.59:0/280512986 >> 172.21.15.59:0/280512986 conn(0x7f05400f84b0 msgr2=0x7f05400fa8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:43.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.218+0000 7f0547aec700 1 -- 172.21.15.59:0/280512986 shutdown_connections 2024-01-23T08:49:43.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.218+0000 7f0547aec700 1 -- 172.21.15.59:0/280512986 wait complete. 2024-01-23T08:49:43.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.219+0000 7f0547aec700 1 Processor -- start 2024-01-23T08:49:43.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.219+0000 7f0547aec700 1 -- start start 2024-01-23T08:49:43.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.219+0000 7f0547aec700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05400fcdd0 0x7f0540071cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:43.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.219+0000 7f0547aec700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05400fdfe0 0x7f0540070170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:43.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.219+0000 7f0547aec700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0540070770 con 0x7f05400fdfe0 2024-01-23T08:49:43.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.219+0000 7f0547aec700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05400708e0 con 0x7f05400fcdd0 2024-01-23T08:49:43.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.219+0000 7f0545087700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05400fdfe0 0x7f0540070170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:43.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.219+0000 7f0545087700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05400fdfe0 0x7f0540070170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34212/0 (socket says 172.21.15.59:34212) 2024-01-23T08:49:43.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.219+0000 7f0545087700 1 -- 172.21.15.59:0/1262605151 learned_addr learned my addr 172.21.15.59:0/1262605151 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:49:43.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.220+0000 7f0545888700 1 --2- 172.21.15.59:0/1262605151 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05400fcdd0 0x7f0540071cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:43.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.220+0000 7f0545087700 1 -- 172.21.15.59:0/1262605151 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05400fcdd0 msgr2=0x7f0540071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:43.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.220+0000 7f0545087700 1 --2- 172.21.15.59:0/1262605151 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05400fcdd0 0x7f0540071cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:43.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.220+0000 7f0545087700 1 -- 172.21.15.59:0/1262605151 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c005040 con 0x7f05400fdfe0 2024-01-23T08:49:43.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.220+0000 7f0545087700 1 --2- 172.21.15.59:0/1262605151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05400fdfe0 0x7f0540070170 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f053c009380 tx=0x7f053c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:43.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.220+0000 7f05367fc700 1 -- 172.21.15.59:0/1262605151 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f053c00e040 con 0x7f05400fdfe0 2024-01-23T08:49:43.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.220+0000 7f05367fc700 1 -- 172.21.15.59:0/1262605151 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f053c00bda0 con 0x7f05400fdfe0 2024-01-23T08:49:43.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.220+0000 7f0547aec700 1 -- 172.21.15.59:0/1262605151 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0540070b60 con 0x7f05400fdfe0 2024-01-23T08:49:43.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.220+0000 7f05367fc700 1 -- 172.21.15.59:0/1262605151 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f053c01c630 con 0x7f05400fdfe0 2024-01-23T08:49:43.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.220+0000 7f0547aec700 1 -- 172.21.15.59:0/1262605151 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05401157d0 con 0x7f05400fdfe0 2024-01-23T08:49:43.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.221+0000 7f0547aec700 1 -- 172.21.15.59:0/1262605151 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05401076e0 con 0x7f05400fdfe0 2024-01-23T08:49:43.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.222+0000 7f05367fc700 1 -- 172.21.15.59:0/1262605151 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f053c0139d0 con 0x7f05400fdfe0 2024-01-23T08:49:43.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.222+0000 7f05367fc700 1 --2- 172.21.15.59:0/1262605151 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f052c068f60 0x7f052c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:43.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.222+0000 7f05367fc700 1 -- 172.21.15.59:0/1262605151 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f053c087df0 con 0x7f05400fdfe0 2024-01-23T08:49:43.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.222+0000 7f0545888700 1 --2- 172.21.15.59:0/1262605151 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f052c068f60 0x7f052c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:43.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.222+0000 7f0545888700 1 --2- 172.21.15.59:0/1262605151 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f052c068f60 0x7f052c06b410 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0530007fd0 tx=0x7f0530009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:43.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.226+0000 7f05367fc700 1 -- 172.21.15.59:0/1262605151 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f053c058130 con 0x7f05400fdfe0 2024-01-23T08:49:43.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.429+0000 7f0547aec700 1 -- 172.21.15.59:0/1262605151 --> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f054002b240 con 0x7f052c068f60 2024-01-23T08:49:43.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.437+0000 7f05367fc700 1 -- 172.21.15.59:0/1262605151 <== mgr.14210 v2:172.21.15.59:6800/3219735162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2682 (secure 0 0 0) 0x7f054002b240 con 0x7f052c068f60 2024-01-23T08:49:43.435 INFO:teuthology.orchestra.run.smithi059.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-01-23T08:49:43.435 INFO:teuthology.orchestra.run.smithi059.stdout:alertmanager.smithi059 smithi059 *:9093,9094 running (3m) 2m ago 5m 26.8M - 0.25.0 c8568f914cd2 1cc540895329 2024-01-23T08:49:43.435 INFO:teuthology.orchestra.run.smithi059.stdout:crash.smithi059 smithi059 running (5m) 2m ago 5m 6656k - 17.2.7-397-gb824e5be 61b360e6ff65 12d41238ee1c 2024-01-23T08:49:43.436 INFO:teuthology.orchestra.run.smithi059.stdout:crash.smithi137 smithi137 running (4m) 68s ago 4m 6665k - 17.2.7-397-gb824e5be 61b360e6ff65 62f889b4ea0f 2024-01-23T08:49:43.436 INFO:teuthology.orchestra.run.smithi059.stdout:grafana.smithi059 smithi059 *:3000 running (3m) 2m ago 4m 82.3M - 9.4.7 954c08fa6188 f2e5dfb17ced 2024-01-23T08:49:43.436 INFO:teuthology.orchestra.run.smithi059.stdout:mgr.smithi059.bzqirf smithi059 *:9283 running (6m) 2m ago 6m 470M - 17.2.7-397-gb824e5be 61b360e6ff65 88cf77d2510a 2024-01-23T08:49:43.436 INFO:teuthology.orchestra.run.smithi059.stdout:mgr.smithi137.ooirrh smithi137 *:8443,9283 running (4m) 68s ago 4m 431M - 17.2.7-397-gb824e5be 61b360e6ff65 5af177b46b24 2024-01-23T08:49:43.436 INFO:teuthology.orchestra.run.smithi059.stdout:mon.smithi059 smithi059 running (6m) 2m ago 6m 41.5M 2048M 17.2.7-397-gb824e5be 61b360e6ff65 810a40012e10 2024-01-23T08:49:43.436 INFO:teuthology.orchestra.run.smithi059.stdout:mon.smithi137 smithi137 running (4m) 68s ago 4m 38.6M 2048M 17.2.7-397-gb824e5be 61b360e6ff65 e1d4c945bf49 2024-01-23T08:49:43.436 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter.smithi059 smithi059 *:9100 running (5m) 2m ago 5m 16.2M - 1.5.0 0da6a335fe13 89ec78f972e4 2024-01-23T08:49:43.436 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter.smithi137 smithi137 *:9100 running (4m) 68s ago 4m 15.7M - 1.5.0 0da6a335fe13 1b38a323818a 2024-01-23T08:49:43.436 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 smithi059 running (3m) 2m ago 3m 36.8M 4289M 17.2.7-397-gb824e5be 61b360e6ff65 11074a2f09ab 2024-01-23T08:49:43.436 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 smithi059 running (2m) 2m ago 2m 38.0M 4289M 17.2.7-397-gb824e5be 61b360e6ff65 efe49a487df2 2024-01-23T08:49:43.436 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 smithi059 running (2m) 2m ago 2m 38.6M 4289M 17.2.7-397-gb824e5be 61b360e6ff65 0e2f974c19f0 2024-01-23T08:49:43.436 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 smithi137 running (2m) 68s ago 2m 39.2M 5312M 17.2.7-397-gb824e5be 61b360e6ff65 a7de37e3a235 2024-01-23T08:49:43.436 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 smithi137 running (99s) 68s ago 98s 38.0M 5312M 17.2.7-397-gb824e5be 61b360e6ff65 017510d7e50d 2024-01-23T08:49:43.437 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 smithi137 running (73s) 68s ago 73s 11.0M 5312M 17.2.7-397-gb824e5be 61b360e6ff65 c37971a0dbb9 2024-01-23T08:49:43.437 INFO:teuthology.orchestra.run.smithi059.stdout:prometheus.smithi059 smithi059 *:9095 running (3m) 2m ago 4m 37.5M - 2.43.0 a07b618ecd1d 21db8bea28d9 2024-01-23T08:49:43.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.439+0000 7f0547aec700 1 -- 172.21.15.59:0/1262605151 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f052c068f60 msgr2=0x7f052c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:43.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.439+0000 7f0547aec700 1 --2- 172.21.15.59:0/1262605151 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f052c068f60 0x7f052c06b410 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0530007fd0 tx=0x7f0530009040 comp rx=0 tx=0).stop 2024-01-23T08:49:43.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.439+0000 7f0547aec700 1 -- 172.21.15.59:0/1262605151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05400fdfe0 msgr2=0x7f0540070170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:43.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.439+0000 7f0547aec700 1 --2- 172.21.15.59:0/1262605151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05400fdfe0 0x7f0540070170 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f053c009380 tx=0x7f053c009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:43.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.439+0000 7f0547aec700 1 -- 172.21.15.59:0/1262605151 shutdown_connections 2024-01-23T08:49:43.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.439+0000 7f0547aec700 1 --2- 172.21.15.59:0/1262605151 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f052c068f60 0x7f052c06b410 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:43.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.439+0000 7f0547aec700 1 --2- 172.21.15.59:0/1262605151 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05400fcdd0 0x7f0540071cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:43.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.439+0000 7f0547aec700 1 --2- 172.21.15.59:0/1262605151 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05400fdfe0 0x7f0540070170 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:43.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.439+0000 7f0547aec700 1 -- 172.21.15.59:0/1262605151 >> 172.21.15.59:0/1262605151 conn(0x7f05400f84b0 msgr2=0x7f05400fa850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:43.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.439+0000 7f0547aec700 1 -- 172.21.15.59:0/1262605151 shutdown_connections 2024-01-23T08:49:43.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:43.439+0000 7f0547aec700 1 -- 172.21.15.59:0/1262605151 wait complete. 2024-01-23T08:49:43.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:43 smithi059 ceph-mon[28424]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:43.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:43 smithi137 ceph-mon[35787]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:44.055 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- bash -c 'ceph orch ls' 2024-01-23T08:49:44.316 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:49:44.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:44 smithi059 ceph-mon[28424]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:49:44.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:44 smithi137 ceph-mon[35787]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:49:45.613 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:45 smithi059 ceph-mon[28424]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:45.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:45 smithi137 ceph-mon[35787]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:45.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.825+0000 7fbc988d8700 1 -- 172.21.15.59:0/183873782 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc940fcf70 msgr2=0x7fbc940fd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:45.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.825+0000 7fbc988d8700 1 --2- 172.21.15.59:0/183873782 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc940fcf70 0x7fbc940fd3e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbc84004940 tx=0x7fbc84009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:45.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.825+0000 7fbc988d8700 1 -- 172.21.15.59:0/183873782 shutdown_connections 2024-01-23T08:49:45.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.825+0000 7fbc988d8700 1 --2- 172.21.15.59:0/183873782 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc940fcf70 0x7fbc940fd3e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:45.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.825+0000 7fbc988d8700 1 --2- 172.21.15.59:0/183873782 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc940fbcc0 0x7fbc940fc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:45.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.825+0000 7fbc988d8700 1 -- 172.21.15.59:0/183873782 >> 172.21.15.59:0/183873782 conn(0x7fbc940f7420 msgr2=0x7fbc940f9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:45.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.825+0000 7fbc988d8700 1 -- 172.21.15.59:0/183873782 shutdown_connections 2024-01-23T08:49:45.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.825+0000 7fbc988d8700 1 -- 172.21.15.59:0/183873782 wait complete. 2024-01-23T08:49:45.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.826+0000 7fbc988d8700 1 Processor -- start 2024-01-23T08:49:45.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.826+0000 7fbc988d8700 1 -- start start 2024-01-23T08:49:45.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.826+0000 7fbc988d8700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc940fbcc0 0x7fbc9410c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:45.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.826+0000 7fbc988d8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc940fcf70 0x7fbc9410caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:45.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.826+0000 7fbc988d8700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9410d0f0 con 0x7fbc940fcf70 2024-01-23T08:49:45.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.826+0000 7fbc988d8700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9410d990 con 0x7fbc940fbcc0 2024-01-23T08:49:45.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.826+0000 7fbc91d9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc940fcf70 0x7fbc9410caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:45.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.826+0000 7fbc91d9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc940fcf70 0x7fbc9410caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34230/0 (socket says 172.21.15.59:34230) 2024-01-23T08:49:45.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.826+0000 7fbc91d9b700 1 -- 172.21.15.59:0/480667812 learned_addr learned my addr 172.21.15.59:0/480667812 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:49:45.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.826+0000 7fbc9259c700 1 --2- 172.21.15.59:0/480667812 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc940fbcc0 0x7fbc9410c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:45.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.827+0000 7fbc91d9b700 1 -- 172.21.15.59:0/480667812 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc940fbcc0 msgr2=0x7fbc9410c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:45.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.827+0000 7fbc91d9b700 1 --2- 172.21.15.59:0/480667812 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc940fbcc0 0x7fbc9410c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:45.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.827+0000 7fbc91d9b700 1 -- 172.21.15.59:0/480667812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc84005040 con 0x7fbc940fcf70 2024-01-23T08:49:45.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.827+0000 7fbc9259c700 1 --2- 172.21.15.59:0/480667812 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc940fbcc0 0x7fbc9410c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-23T08:49:45.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.827+0000 7fbc91d9b700 1 --2- 172.21.15.59:0/480667812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc940fcf70 0x7fbc9410caf0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbc84009360 tx=0x7fbc84009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:45.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.827+0000 7fbc82ffd700 1 -- 172.21.15.59:0/480667812 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc8400e040 con 0x7fbc940fcf70 2024-01-23T08:49:45.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.827+0000 7fbc988d8700 1 -- 172.21.15.59:0/480667812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc9410dc10 con 0x7fbc940fcf70 2024-01-23T08:49:45.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.827+0000 7fbc82ffd700 1 -- 172.21.15.59:0/480667812 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc8400bcd0 con 0x7fbc940fcf70 2024-01-23T08:49:45.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.827+0000 7fbc988d8700 1 -- 172.21.15.59:0/480667812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc9410e150 con 0x7fbc940fcf70 2024-01-23T08:49:45.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.827+0000 7fbc82ffd700 1 -- 172.21.15.59:0/480667812 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc8401c630 con 0x7fbc940fcf70 2024-01-23T08:49:45.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.828+0000 7fbc988d8700 1 -- 172.21.15.59:0/480667812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc9404bd60 con 0x7fbc940fcf70 2024-01-23T08:49:45.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.828+0000 7fbc82ffd700 1 -- 172.21.15.59:0/480667812 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fbc84013960 con 0x7fbc940fcf70 2024-01-23T08:49:45.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.829+0000 7fbc82ffd700 1 --2- 172.21.15.59:0/480667812 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbc7c068f60 0x7fbc7c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:45.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.829+0000 7fbc82ffd700 1 -- 172.21.15.59:0/480667812 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbc84087f90 con 0x7fbc940fcf70 2024-01-23T08:49:45.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.829+0000 7fbc9259c700 1 --2- 172.21.15.59:0/480667812 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbc7c068f60 0x7fbc7c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:45.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.829+0000 7fbc9259c700 1 --2- 172.21.15.59:0/480667812 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbc7c068f60 0x7fbc7c06b410 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbc88009fd0 tx=0x7fbc88009ee0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:45.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:45.834+0000 7fbc82ffd700 1 -- 172.21.15.59:0/480667812 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fbc84058190 con 0x7fbc940fcf70 2024-01-23T08:49:46.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:46.032+0000 7fbc988d8700 1 -- 172.21.15.59:0/480667812 --> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc9410e590 con 0x7fbc7c068f60 2024-01-23T08:49:46.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:46.037+0000 7fbc82ffd700 1 -- 172.21.15.59:0/480667812 <== mgr.14210 v2:172.21.15.59:6800/3219735162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1170 (secure 0 0 0) 0x7fbc9410e590 con 0x7fbc7c068f60 2024-01-23T08:49:46.034 INFO:teuthology.orchestra.run.smithi059.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-01-23T08:49:46.035 INFO:teuthology.orchestra.run.smithi059.stdout:alertmanager ?:9093,9094 1/1 2m ago 5m count:1 2024-01-23T08:49:46.035 INFO:teuthology.orchestra.run.smithi059.stdout:crash 2/2 2m ago 6m * 2024-01-23T08:49:46.035 INFO:teuthology.orchestra.run.smithi059.stdout:grafana ?:3000 1/1 2m ago 5m count:1 2024-01-23T08:49:46.035 INFO:teuthology.orchestra.run.smithi059.stdout:mgr 2/2 2m ago 6m count:2 2024-01-23T08:49:46.035 INFO:teuthology.orchestra.run.smithi059.stdout:mon 2/2 2m ago 5m smithi059:172.21.15.59=smithi059;smithi137:172.21.15.137=smithi137;count:2 2024-01-23T08:49:46.035 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter ?:9100 2/2 2m ago 5m * 2024-01-23T08:49:46.035 INFO:teuthology.orchestra.run.smithi059.stdout:osd 6 2m ago - 2024-01-23T08:49:46.035 INFO:teuthology.orchestra.run.smithi059.stdout:prometheus ?:9095 1/1 2m ago 5m count:1 2024-01-23T08:49:46.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:46.039+0000 7fbc988d8700 1 -- 172.21.15.59:0/480667812 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbc7c068f60 msgr2=0x7fbc7c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:46.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:46.039+0000 7fbc988d8700 1 --2- 172.21.15.59:0/480667812 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbc7c068f60 0x7fbc7c06b410 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbc88009fd0 tx=0x7fbc88009ee0 comp rx=0 tx=0).stop 2024-01-23T08:49:46.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:46.039+0000 7fbc988d8700 1 -- 172.21.15.59:0/480667812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc940fcf70 msgr2=0x7fbc9410caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:46.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:46.039+0000 7fbc988d8700 1 --2- 172.21.15.59:0/480667812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc940fcf70 0x7fbc9410caf0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbc84009360 tx=0x7fbc84009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:46.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:46.039+0000 7fbc988d8700 1 -- 172.21.15.59:0/480667812 shutdown_connections 2024-01-23T08:49:46.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:46.039+0000 7fbc988d8700 1 --2- 172.21.15.59:0/480667812 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbc7c068f60 0x7fbc7c06b410 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:46.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:46.039+0000 7fbc988d8700 1 --2- 172.21.15.59:0/480667812 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc940fbcc0 0x7fbc9410c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:46.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:46.039+0000 7fbc988d8700 1 --2- 172.21.15.59:0/480667812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc940fcf70 0x7fbc9410caf0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:46.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:46.039+0000 7fbc988d8700 1 -- 172.21.15.59:0/480667812 >> 172.21.15.59:0/480667812 conn(0x7fbc940f7420 msgr2=0x7fbc941001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:46.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:46.039+0000 7fbc988d8700 1 -- 172.21.15.59:0/480667812 shutdown_connections 2024-01-23T08:49:46.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:46.039+0000 7fbc988d8700 1 -- 172.21.15.59:0/480667812 wait complete. 2024-01-23T08:49:46.658 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- bash -c 'ceph orch host ls' 2024-01-23T08:49:46.917 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:49:47.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:47 smithi059 ceph-mon[28424]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:49:47.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:47 smithi059 ceph-mon[28424]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:47.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:47 smithi137 ceph-mon[35787]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:49:47.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:47 smithi137 ceph-mon[35787]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:48.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.516+0000 7f2fcb197700 1 -- 172.21.15.59:0/1760503714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fc40fd910 msgr2=0x7f2fc40ffcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:48.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.516+0000 7f2fcb197700 1 --2- 172.21.15.59:0/1760503714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fc40fd910 0x7f2fc40ffcf0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4004940 tx=0x7f2fb4009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:48.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.516+0000 7f2fcb197700 1 -- 172.21.15.59:0/1760503714 shutdown_connections 2024-01-23T08:49:48.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.516+0000 7f2fcb197700 1 --2- 172.21.15.59:0/1760503714 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fc4100230 0x7f2fc4102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:48.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.516+0000 7f2fcb197700 1 --2- 172.21.15.59:0/1760503714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fc40fd910 0x7f2fc40ffcf0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:48.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.516+0000 7f2fcb197700 1 -- 172.21.15.59:0/1760503714 >> 172.21.15.59:0/1760503714 conn(0x7f2fc40f73b0 msgr2=0x7f2fc40f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:48.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.516+0000 7f2fcb197700 1 -- 172.21.15.59:0/1760503714 shutdown_connections 2024-01-23T08:49:48.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.516+0000 7f2fcb197700 1 -- 172.21.15.59:0/1760503714 wait complete. 2024-01-23T08:49:48.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.516+0000 7f2fcb197700 1 Processor -- start 2024-01-23T08:49:48.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.516+0000 7f2fcb197700 1 -- start start 2024-01-23T08:49:48.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.517+0000 7f2fcb197700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fc40fd910 0x7f2fc410d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:48.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.517+0000 7f2fcb197700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fc4100230 0x7f2fc410dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:48.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.517+0000 7f2fcb197700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc410e290 con 0x7f2fc40fd910 2024-01-23T08:49:48.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.517+0000 7f2fcb197700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc410ead0 con 0x7f2fc4100230 2024-01-23T08:49:48.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.517+0000 7f2fc8f33700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fc40fd910 0x7f2fc410d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:48.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.517+0000 7f2fc8f33700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fc40fd910 0x7f2fc410d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57006/0 (socket says 172.21.15.59:57006) 2024-01-23T08:49:48.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.517+0000 7f2fc8f33700 1 -- 172.21.15.59:0/2353876560 learned_addr learned my addr 172.21.15.59:0/2353876560 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:49:48.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.517+0000 7f2fc3fff700 1 --2- 172.21.15.59:0/2353876560 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fc4100230 0x7f2fc410dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:48.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.517+0000 7f2fc8f33700 1 -- 172.21.15.59:0/2353876560 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fc4100230 msgr2=0x7f2fc410dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:48.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.517+0000 7f2fc8f33700 1 --2- 172.21.15.59:0/2353876560 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fc4100230 0x7f2fc410dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:48.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.517+0000 7f2fc8f33700 1 -- 172.21.15.59:0/2353876560 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb4005040 con 0x7f2fc40fd910 2024-01-23T08:49:48.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.517+0000 7f2fc8f33700 1 --2- 172.21.15.59:0/2353876560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fc40fd910 0x7f2fc410d6c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4009380 tx=0x7f2fb4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:48.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.517+0000 7f2fc17fa700 1 -- 172.21.15.59:0/2353876560 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fb400e040 con 0x7f2fc40fd910 2024-01-23T08:49:48.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.517+0000 7f2fc17fa700 1 -- 172.21.15.59:0/2353876560 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2fb400bda0 con 0x7f2fc40fd910 2024-01-23T08:49:48.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.517+0000 7f2fcb197700 1 -- 172.21.15.59:0/2353876560 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc410ed50 con 0x7f2fc40fd910 2024-01-23T08:49:48.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.518+0000 7f2fc17fa700 1 -- 172.21.15.59:0/2353876560 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fb401c630 con 0x7f2fc40fd910 2024-01-23T08:49:48.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.518+0000 7f2fcb197700 1 -- 172.21.15.59:0/2353876560 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc410f290 con 0x7f2fc40fd910 2024-01-23T08:49:48.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.518+0000 7f2fcb197700 1 -- 172.21.15.59:0/2353876560 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc404bd60 con 0x7f2fc40fd910 2024-01-23T08:49:48.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.519+0000 7f2fc17fa700 1 -- 172.21.15.59:0/2353876560 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f2fb40139d0 con 0x7f2fc40fd910 2024-01-23T08:49:48.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.519+0000 7f2fc17fa700 1 --2- 172.21.15.59:0/2353876560 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f2fac068f60 0x7f2fac06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:48.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.519+0000 7f2fc17fa700 1 -- 172.21.15.59:0/2353876560 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2fb4087f10 con 0x7f2fc40fd910 2024-01-23T08:49:48.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.519+0000 7f2fc3fff700 1 --2- 172.21.15.59:0/2353876560 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f2fac068f60 0x7f2fac06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:48.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.520+0000 7f2fc3fff700 1 --2- 172.21.15.59:0/2353876560 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f2fac068f60 0x7f2fac06b410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8006fd0 tx=0x7f2fb8008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:48.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.524+0000 7f2fc17fa700 1 -- 172.21.15.59:0/2353876560 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2fb40581b0 con 0x7f2fc40fd910 2024-01-23T08:49:48.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.721+0000 7f2fcb197700 1 -- 172.21.15.59:0/2353876560 --> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f2fc410f6b0 con 0x7f2fac068f60 2024-01-23T08:49:48.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.722+0000 7f2fc17fa700 1 -- 172.21.15.59:0/2353876560 <== mgr.14210 v2:172.21.15.59:6800/3219735162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f2fc410f6b0 con 0x7f2fac068f60 2024-01-23T08:49:48.719 INFO:teuthology.orchestra.run.smithi059.stdout:HOST ADDR LABELS STATUS 2024-01-23T08:49:48.719 INFO:teuthology.orchestra.run.smithi059.stdout:smithi059 172.21.15.59 2024-01-23T08:49:48.719 INFO:teuthology.orchestra.run.smithi059.stdout:smithi137 172.21.15.137 2024-01-23T08:49:48.720 INFO:teuthology.orchestra.run.smithi059.stdout:2 hosts in cluster 2024-01-23T08:49:48.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.725+0000 7f2fcb197700 1 -- 172.21.15.59:0/2353876560 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f2fac068f60 msgr2=0x7f2fac06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:48.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.725+0000 7f2fcb197700 1 --2- 172.21.15.59:0/2353876560 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f2fac068f60 0x7f2fac06b410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8006fd0 tx=0x7f2fb8008040 comp rx=0 tx=0).stop 2024-01-23T08:49:48.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.725+0000 7f2fcb197700 1 -- 172.21.15.59:0/2353876560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fc40fd910 msgr2=0x7f2fc410d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:48.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.725+0000 7f2fcb197700 1 --2- 172.21.15.59:0/2353876560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fc40fd910 0x7f2fc410d6c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4009380 tx=0x7f2fb4009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:48.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.725+0000 7f2fcb197700 1 -- 172.21.15.59:0/2353876560 shutdown_connections 2024-01-23T08:49:48.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.725+0000 7f2fcb197700 1 --2- 172.21.15.59:0/2353876560 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f2fac068f60 0x7f2fac06b410 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:48.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.725+0000 7f2fcb197700 1 --2- 172.21.15.59:0/2353876560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fc40fd910 0x7f2fc410d6c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:48.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.725+0000 7f2fcb197700 1 --2- 172.21.15.59:0/2353876560 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fc4100230 0x7f2fc410dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:48.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.725+0000 7f2fcb197700 1 -- 172.21.15.59:0/2353876560 >> 172.21.15.59:0/2353876560 conn(0x7f2fc40f73b0 msgr2=0x7f2fc40f9790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:48.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.725+0000 7f2fcb197700 1 -- 172.21.15.59:0/2353876560 shutdown_connections 2024-01-23T08:49:48.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:48.725+0000 7f2fcb197700 1 -- 172.21.15.59:0/2353876560 wait complete. 2024-01-23T08:49:49.417 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- bash -c 'ceph orch device ls' 2024-01-23T08:49:49.675 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:49:49.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:49 smithi059 ceph-mon[28424]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:49.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:49 smithi137 ceph-mon[35787]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:50.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:50 smithi059 ceph-mon[28424]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:49:50.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:50 smithi137 ceph-mon[35787]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:49:51.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.424+0000 7f382957b700 1 -- 172.21.15.59:0/4214593585 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3824066b00 msgr2=0x7f38240ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:51.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.424+0000 7f382957b700 1 --2- 172.21.15.59:0/4214593585 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3824066b00 0x7f38240ff980 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3814004940 tx=0x7f3814009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:51.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.424+0000 7f382957b700 1 -- 172.21.15.59:0/4214593585 shutdown_connections 2024-01-23T08:49:51.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.424+0000 7f382957b700 1 --2- 172.21.15.59:0/4214593585 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38240fff50 0x7f38241023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:51.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.424+0000 7f382957b700 1 --2- 172.21.15.59:0/4214593585 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3824066b00 0x7f38240ff980 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:51.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.424+0000 7f382957b700 1 -- 172.21.15.59:0/4214593585 >> 172.21.15.59:0/4214593585 conn(0x7f38240f7390 msgr2=0x7f38240f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:51.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.424+0000 7f382957b700 1 -- 172.21.15.59:0/4214593585 shutdown_connections 2024-01-23T08:49:51.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.424+0000 7f382957b700 1 -- 172.21.15.59:0/4214593585 wait complete. 2024-01-23T08:49:51.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.424+0000 7f382957b700 1 Processor -- start 2024-01-23T08:49:51.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.424+0000 7f382957b700 1 -- start start 2024-01-23T08:49:51.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.425+0000 7f382957b700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3824066b00 0x7f382410d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:51.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.425+0000 7f382957b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38240fff50 0x7f382410dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:51.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.425+0000 7f382957b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382410e380 con 0x7f38240fff50 2024-01-23T08:49:51.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.425+0000 7f382957b700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382410ec20 con 0x7f3824066b00 2024-01-23T08:49:51.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.425+0000 7f381a7fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38240fff50 0x7f382410dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:51.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.425+0000 7f381a7fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38240fff50 0x7f382410dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57012/0 (socket says 172.21.15.59:57012) 2024-01-23T08:49:51.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.425+0000 7f381a7fc700 1 -- 172.21.15.59:0/962564865 learned_addr learned my addr 172.21.15.59:0/962564865 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:49:51.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.425+0000 7f3822ffd700 1 --2- 172.21.15.59:0/962564865 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3824066b00 0x7f382410d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:51.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.425+0000 7f381a7fc700 1 -- 172.21.15.59:0/962564865 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3824066b00 msgr2=0x7f382410d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:51.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.425+0000 7f381a7fc700 1 --2- 172.21.15.59:0/962564865 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3824066b00 0x7f382410d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:51.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.425+0000 7f381a7fc700 1 -- 172.21.15.59:0/962564865 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3814005040 con 0x7f38240fff50 2024-01-23T08:49:51.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.425+0000 7f381a7fc700 1 --2- 172.21.15.59:0/962564865 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38240fff50 0x7f382410dd80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f380c002a50 tx=0x7f380c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:51.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.425+0000 7f381bfff700 1 -- 172.21.15.59:0/962564865 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f380c007cc0 con 0x7f38240fff50 2024-01-23T08:49:51.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.425+0000 7f381bfff700 1 -- 172.21.15.59:0/962564865 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f380c00f040 con 0x7f38240fff50 2024-01-23T08:49:51.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.425+0000 7f382957b700 1 -- 172.21.15.59:0/962564865 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f382410ef00 con 0x7f38240fff50 2024-01-23T08:49:51.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.425+0000 7f381bfff700 1 -- 172.21.15.59:0/962564865 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f380c00d920 con 0x7f38240fff50 2024-01-23T08:49:51.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.425+0000 7f382957b700 1 -- 172.21.15.59:0/962564865 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f382410f440 con 0x7f38240fff50 2024-01-23T08:49:51.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.426+0000 7f382957b700 1 -- 172.21.15.59:0/962564865 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38241076f0 con 0x7f38240fff50 2024-01-23T08:49:51.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.427+0000 7f381bfff700 1 -- 172.21.15.59:0/962564865 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f380c009680 con 0x7f38240fff50 2024-01-23T08:49:51.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.427+0000 7f381bfff700 1 --2- 172.21.15.59:0/962564865 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f3810068f60 0x7f381006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:51.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.427+0000 7f381bfff700 1 -- 172.21.15.59:0/962564865 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f380c0867d0 con 0x7f38240fff50 2024-01-23T08:49:51.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.427+0000 7f3822ffd700 1 --2- 172.21.15.59:0/962564865 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f3810068f60 0x7f381006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:51.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.428+0000 7f3822ffd700 1 --2- 172.21.15.59:0/962564865 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f3810068f60 0x7f381006b410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f381400b3b0 tx=0x7f3814009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:51.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.432+0000 7f381bfff700 1 -- 172.21.15.59:0/962564865 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f380c0569f0 con 0x7f38240fff50 2024-01-23T08:49:51.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:51 smithi059 ceph-mon[28424]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:51.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.631+0000 7f382957b700 1 -- 172.21.15.59:0/962564865 --> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f382410f660 con 0x7f3810068f60 2024-01-23T08:49:51.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.632+0000 7f381bfff700 1 -- 172.21.15.59:0/962564865 <== mgr.14210 v2:172.21.15.59:6800/3219735162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f382410f660 con 0x7f3810068f60 2024-01-23T08:49:51.629 INFO:teuthology.orchestra.run.smithi059.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-01-23T08:49:51.630 INFO:teuthology.orchestra.run.smithi059.stdout:smithi059 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330009L400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-01-23T08:49:51.630 INFO:teuthology.orchestra.run.smithi059.stdout:smithi137 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233004A400BGN 372G No 73s ago Has a FileSystem, LVM detected 2024-01-23T08:49:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.634+0000 7f382957b700 1 -- 172.21.15.59:0/962564865 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f3810068f60 msgr2=0x7f381006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.634+0000 7f382957b700 1 --2- 172.21.15.59:0/962564865 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f3810068f60 0x7f381006b410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f381400b3b0 tx=0x7f3814009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:51.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.634+0000 7f382957b700 1 -- 172.21.15.59:0/962564865 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38240fff50 msgr2=0x7f382410dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:51.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.634+0000 7f382957b700 1 --2- 172.21.15.59:0/962564865 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38240fff50 0x7f382410dd80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f380c002a50 tx=0x7f380c004600 comp rx=0 tx=0).stop 2024-01-23T08:49:51.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.634+0000 7f382957b700 1 -- 172.21.15.59:0/962564865 shutdown_connections 2024-01-23T08:49:51.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.634+0000 7f382957b700 1 --2- 172.21.15.59:0/962564865 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f3810068f60 0x7f381006b410 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:51.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.634+0000 7f382957b700 1 --2- 172.21.15.59:0/962564865 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3824066b00 0x7f382410d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:51.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.634+0000 7f382957b700 1 --2- 172.21.15.59:0/962564865 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38240fff50 0x7f382410dd80 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:51.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.634+0000 7f382957b700 1 -- 172.21.15.59:0/962564865 >> 172.21.15.59:0/962564865 conn(0x7f38240f7390 msgr2=0x7f38240f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:51.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.635+0000 7f382957b700 1 -- 172.21.15.59:0/962564865 shutdown_connections 2024-01-23T08:49:51.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:51.635+0000 7f382957b700 1 -- 172.21.15.59:0/962564865 wait complete. 2024-01-23T08:49:51.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:51 smithi137 ceph-mon[35787]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:52.300 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-01-23T08:49:52.323 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2024-01-23T08:49:52.324 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-01-23T08:49:52.591 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:49:52.713 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:52 smithi059 ceph-mon[28424]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:49:52.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:52 smithi137 ceph-mon[35787]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:49:53.702 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:53 smithi059 ceph-mon[28424]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:53.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:53 smithi137 ceph-mon[35787]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:54.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.338+0000 7f6fb7043700 1 -- 172.21.15.59:0/1565014485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb00fffb0 msgr2=0x7f6fb01023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:54.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.338+0000 7f6fb7043700 1 --2- 172.21.15.59:0/1565014485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb00fffb0 0x7f6fb01023b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c004940 tx=0x7f6f9c009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:54.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.338+0000 7f6fb7043700 1 -- 172.21.15.59:0/1565014485 shutdown_connections 2024-01-23T08:49:54.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.338+0000 7f6fb7043700 1 --2- 172.21.15.59:0/1565014485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb00fffb0 0x7f6fb01023b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:54.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.338+0000 7f6fb7043700 1 --2- 172.21.15.59:0/1565014485 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fb0066aa0 0x7f6fb00ff9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:54.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.338+0000 7f6fb7043700 1 -- 172.21.15.59:0/1565014485 >> 172.21.15.59:0/1565014485 conn(0x7f6fb00f7390 msgr2=0x7f6fb00f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:54.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.338+0000 7f6fb7043700 1 -- 172.21.15.59:0/1565014485 shutdown_connections 2024-01-23T08:49:54.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.338+0000 7f6fb7043700 1 -- 172.21.15.59:0/1565014485 wait complete. 2024-01-23T08:49:54.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.339+0000 7f6fb7043700 1 Processor -- start 2024-01-23T08:49:54.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.339+0000 7f6fb7043700 1 -- start start 2024-01-23T08:49:54.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.339+0000 7f6fb7043700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fb0066aa0 0x7f6fb010d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:54.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.339+0000 7f6fb7043700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb00fffb0 0x7f6fb010dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:54.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.339+0000 7f6fb7043700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb010e2c0 con 0x7f6fb00fffb0 2024-01-23T08:49:54.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.339+0000 7f6fb7043700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb010eb00 con 0x7f6fb0066aa0 2024-01-23T08:49:54.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.339+0000 7f6faffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb00fffb0 0x7f6fb010dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:54.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.339+0000 7f6faffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb00fffb0 0x7f6fb010dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57034/0 (socket says 172.21.15.59:57034) 2024-01-23T08:49:54.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.339+0000 7f6faffff700 1 -- 172.21.15.59:0/4196415161 learned_addr learned my addr 172.21.15.59:0/4196415161 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:49:54.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.339+0000 7f6fb4ddf700 1 --2- 172.21.15.59:0/4196415161 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fb0066aa0 0x7f6fb010d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:54.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.340+0000 7f6faffff700 1 -- 172.21.15.59:0/4196415161 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fb0066aa0 msgr2=0x7f6fb010d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:54.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.340+0000 7f6faffff700 1 --2- 172.21.15.59:0/4196415161 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fb0066aa0 0x7f6fb010d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:54.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.340+0000 7f6faffff700 1 -- 172.21.15.59:0/4196415161 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9c005040 con 0x7f6fb00fffb0 2024-01-23T08:49:54.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.340+0000 7f6faffff700 1 --2- 172.21.15.59:0/4196415161 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb00fffb0 0x7f6fb010dc30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c004f90 tx=0x7f6f9c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:54.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.340+0000 7f6fad7fa700 1 -- 172.21.15.59:0/4196415161 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f9c00e040 con 0x7f6fb00fffb0 2024-01-23T08:49:54.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.340+0000 7f6fad7fa700 1 -- 172.21.15.59:0/4196415161 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f9c00bc50 con 0x7f6fb00fffb0 2024-01-23T08:49:54.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.340+0000 7f6fad7fa700 1 -- 172.21.15.59:0/4196415161 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f9c01c630 con 0x7f6fb00fffb0 2024-01-23T08:49:54.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.340+0000 7f6fb7043700 1 -- 172.21.15.59:0/4196415161 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb010ed80 con 0x7f6fb00fffb0 2024-01-23T08:49:54.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.340+0000 7f6fb7043700 1 -- 172.21.15.59:0/4196415161 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb010f2c0 con 0x7f6fb00fffb0 2024-01-23T08:49:54.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.341+0000 7f6fb7043700 1 -- 172.21.15.59:0/4196415161 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb004bd60 con 0x7f6fb00fffb0 2024-01-23T08:49:54.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.341+0000 7f6fad7fa700 1 -- 172.21.15.59:0/4196415161 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f6f9c013420 con 0x7f6fb00fffb0 2024-01-23T08:49:54.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.342+0000 7f6fad7fa700 1 --2- 172.21.15.59:0/4196415161 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f6fa0068f60 0x7f6fa006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:49:54.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.342+0000 7f6fad7fa700 1 -- 172.21.15.59:0/4196415161 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6f9c087e10 con 0x7f6fb00fffb0 2024-01-23T08:49:54.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.342+0000 7f6fb4ddf700 1 --2- 172.21.15.59:0/4196415161 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f6fa0068f60 0x7f6fa006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:49:54.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.342+0000 7f6fb4ddf700 1 --2- 172.21.15.59:0/4196415161 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f6fa0068f60 0x7f6fa006b410 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4009bf0 tx=0x7f6fa40091a0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:49:54.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.346+0000 7f6fad7fa700 1 -- 172.21.15.59:0/4196415161 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6f9c0580b0 con 0x7f6fb00fffb0 2024-01-23T08:49:54.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:54.568+0000 7f6fb7043700 1 -- 172.21.15.59:0/4196415161 --> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fb010f750 con 0x7f6fa0068f60 2024-01-23T08:49:55.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:55 smithi059 ceph-mon[28424]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:55.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:55 smithi059 ceph-mon[28424]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:49:55.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:55 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-01-23T08:49:55.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:55 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:49:55.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:55 smithi137 ceph-mon[35787]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:55.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:55 smithi137 ceph-mon[35787]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:49:55.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:55 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-01-23T08:49:55.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:55 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:49:56.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:56.471+0000 7f6fad7fa700 1 -- 172.21.15.59:0/4196415161 <== mgr.14210 v2:172.21.15.59:6800/3219735162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6fb010f750 con 0x7f6fa0068f60 2024-01-23T08:49:56.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:56.475+0000 7f6fb7043700 1 -- 172.21.15.59:0/4196415161 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f6fa0068f60 msgr2=0x7f6fa006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:56.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:56.475+0000 7f6fb7043700 1 --2- 172.21.15.59:0/4196415161 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f6fa0068f60 0x7f6fa006b410 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4009bf0 tx=0x7f6fa40091a0 comp rx=0 tx=0).stop 2024-01-23T08:49:56.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:56.475+0000 7f6fb7043700 1 -- 172.21.15.59:0/4196415161 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb00fffb0 msgr2=0x7f6fb010dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:49:56.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:56.475+0000 7f6fb7043700 1 --2- 172.21.15.59:0/4196415161 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb00fffb0 0x7f6fb010dc30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c004f90 tx=0x7f6f9c009e30 comp rx=0 tx=0).stop 2024-01-23T08:49:56.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:56.476+0000 7f6fb7043700 1 -- 172.21.15.59:0/4196415161 shutdown_connections 2024-01-23T08:49:56.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:56.476+0000 7f6fb7043700 1 --2- 172.21.15.59:0/4196415161 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f6fa0068f60 0x7f6fa006b410 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:56.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:56.476+0000 7f6fb7043700 1 --2- 172.21.15.59:0/4196415161 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6fb0066aa0 0x7f6fb010d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:56.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:56.476+0000 7f6fb7043700 1 --2- 172.21.15.59:0/4196415161 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb00fffb0 0x7f6fb010dc30 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:49:56.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:56.476+0000 7f6fb7043700 1 -- 172.21.15.59:0/4196415161 >> 172.21.15.59:0/4196415161 conn(0x7f6fb00f7390 msgr2=0x7f6fb00f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:49:56.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:56.476+0000 7f6fb7043700 1 -- 172.21.15.59:0/4196415161 shutdown_connections 2024-01-23T08:49:56.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:49:56.476+0000 7f6fb7043700 1 -- 172.21.15.59:0/4196415161 wait complete. 2024-01-23T08:49:56.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:56 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-01-23T08:49:56.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:56 smithi059 ceph-mon[28424]: osdmap e42: 6 total, 6 up, 6 in 2024-01-23T08:49:56.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:56 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-01-23T08:49:56.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:49:56.449+0000 7f5c94d67700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-23T08:49:56.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:56 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-01-23T08:49:56.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:56 smithi137 ceph-mon[35787]: osdmap e42: 6 total, 6 up, 6 in 2024-01-23T08:49:56.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:56 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-01-23T08:49:57.144 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- bash -c 'ceph fs dump' 2024-01-23T08:49:57.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:57 smithi059 ceph-mon[28424]: pgmap v173: 33 pgs: 10 creating+peering, 22 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:57.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:57 smithi059 ceph-mon[28424]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-01-23T08:49:57.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:57 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-01-23T08:49:57.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:57 smithi059 ceph-mon[28424]: osdmap e43: 6 total, 6 up, 6 in 2024-01-23T08:49:57.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:57 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-01-23T08:49:57.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:57 smithi059 ceph-mon[28424]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-23T08:49:57.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:57 smithi059 ceph-mon[28424]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-01-23T08:49:57.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:57 smithi059 ceph-mon[28424]: osdmap e44: 6 total, 6 up, 6 in 2024-01-23T08:49:57.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:57 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-01-23T08:49:57.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:57 smithi059 ceph-mon[28424]: fsmap cephfs:0 2024-01-23T08:49:57.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:57 smithi059 ceph-mon[28424]: Saving service mds.cephfs spec with placement count:4 2024-01-23T08:49:57.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:57 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:49:57.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:57 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:49:57.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:57 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:49:57.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:57 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:49:57.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:57 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:49:57.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:57 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi059.lsxsmm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-23T08:49:57.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:57 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi059.lsxsmm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-23T08:49:57.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:57 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:49:57.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:57 smithi137 ceph-mon[35787]: pgmap v173: 33 pgs: 10 creating+peering, 22 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:57.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:57 smithi137 ceph-mon[35787]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-01-23T08:49:57.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:57 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-01-23T08:49:57.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:57 smithi137 ceph-mon[35787]: osdmap e43: 6 total, 6 up, 6 in 2024-01-23T08:49:57.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:57 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-01-23T08:49:57.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:57 smithi137 ceph-mon[35787]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-23T08:49:57.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:57 smithi137 ceph-mon[35787]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-01-23T08:49:57.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:57 smithi137 ceph-mon[35787]: osdmap e44: 6 total, 6 up, 6 in 2024-01-23T08:49:57.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:57 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-01-23T08:49:57.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:57 smithi137 ceph-mon[35787]: fsmap cephfs:0 2024-01-23T08:49:57.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:57 smithi137 ceph-mon[35787]: Saving service mds.cephfs spec with placement count:4 2024-01-23T08:49:57.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:57 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:49:57.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:57 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:49:57.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:57 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:49:57.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:57 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:49:57.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:57 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:49:57.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:57 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi059.lsxsmm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-23T08:49:57.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:57 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi059.lsxsmm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-23T08:49:57.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:57 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:49:57.867 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:49:58.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:58 smithi059 ceph-mon[28424]: Deploying daemon mds.cephfs.smithi059.lsxsmm on smithi059 2024-01-23T08:49:58.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:58 smithi059 ceph-mon[28424]: osdmap e45: 6 total, 6 up, 6 in 2024-01-23T08:49:58.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:58 smithi137 ceph-mon[35787]: Deploying daemon mds.cephfs.smithi059.lsxsmm on smithi059 2024-01-23T08:49:58.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:58 smithi137 ceph-mon[35787]: osdmap e45: 6 total, 6 up, 6 in 2024-01-23T08:49:59.517 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:59 smithi059 ceph-mon[28424]: pgmap v177: 65 pgs: 16 creating+peering, 23 unknown, 26 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:59.517 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:59 smithi059 ceph-mon[28424]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-01-23T08:49:59.518 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:49:59 smithi059 ceph-mon[28424]: osdmap e46: 6 total, 6 up, 6 in 2024-01-23T08:49:59.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:59 smithi137 ceph-mon[35787]: pgmap v177: 65 pgs: 16 creating+peering, 23 unknown, 26 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:49:59.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:59 smithi137 ceph-mon[35787]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-01-23T08:49:59.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:49:59 smithi137 ceph-mon[35787]: osdmap e46: 6 total, 6 up, 6 in 2024-01-23T08:50:00.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.041+0000 7f324bb53700 1 -- 172.21.15.59:0/483389064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32440fcee0 msgr2=0x7f32440fd330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:00.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.041+0000 7f324bb53700 1 --2- 172.21.15.59:0/483389064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32440fcee0 0x7f32440fd330 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3234004940 tx=0x7f3234009e30 comp rx=0 tx=0).stop 2024-01-23T08:50:00.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.041+0000 7f324bb53700 1 -- 172.21.15.59:0/483389064 shutdown_connections 2024-01-23T08:50:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.041+0000 7f324bb53700 1 --2- 172.21.15.59:0/483389064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32440fcee0 0x7f32440fd330 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.041+0000 7f324bb53700 1 --2- 172.21.15.59:0/483389064 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32440fbcd0 0x7f32440fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.041+0000 7f324bb53700 1 -- 172.21.15.59:0/483389064 >> 172.21.15.59:0/483389064 conn(0x7f32440f7370 msgr2=0x7f32440f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.042+0000 7f324bb53700 1 -- 172.21.15.59:0/483389064 shutdown_connections 2024-01-23T08:50:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.042+0000 7f324bb53700 1 -- 172.21.15.59:0/483389064 wait complete. 2024-01-23T08:50:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.042+0000 7f324bb53700 1 Processor -- start 2024-01-23T08:50:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.042+0000 7f324bb53700 1 -- start start 2024-01-23T08:50:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.042+0000 7f324bb53700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32440fbcd0 0x7f324410c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.042+0000 7f324bb53700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32440fcee0 0x7f324410cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.042+0000 7f324bb53700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324410d190 con 0x7f32440fcee0 2024-01-23T08:50:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.042+0000 7f324bb53700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324410da30 con 0x7f32440fbcd0 2024-01-23T08:50:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.043+0000 7f32490ee700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32440fcee0 0x7f324410cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.043+0000 7f32490ee700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32440fcee0 0x7f324410cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56336/0 (socket says 172.21.15.59:56336) 2024-01-23T08:50:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.043+0000 7f32490ee700 1 -- 172.21.15.59:0/2394864545 learned_addr learned my addr 172.21.15.59:0/2394864545 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:50:00.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.043+0000 7f32498ef700 1 --2- 172.21.15.59:0/2394864545 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32440fbcd0 0x7f324410c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:00.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.043+0000 7f32490ee700 1 -- 172.21.15.59:0/2394864545 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32440fbcd0 msgr2=0x7f324410c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:00.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.043+0000 7f32490ee700 1 --2- 172.21.15.59:0/2394864545 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32440fbcd0 0x7f324410c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:00.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.043+0000 7f32490ee700 1 -- 172.21.15.59:0/2394864545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3234005040 con 0x7f32440fcee0 2024-01-23T08:50:00.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.043+0000 7f32490ee700 1 --2- 172.21.15.59:0/2394864545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32440fcee0 0x7f324410cb90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f32340093b0 tx=0x7f3234009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:00.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.044+0000 7f323a7fc700 1 -- 172.21.15.59:0/2394864545 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f323400e040 con 0x7f32440fcee0 2024-01-23T08:50:00.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.044+0000 7f323a7fc700 1 -- 172.21.15.59:0/2394864545 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3234004290 con 0x7f32440fcee0 2024-01-23T08:50:00.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.044+0000 7f324bb53700 1 -- 172.21.15.59:0/2394864545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f324410dcb0 con 0x7f32440fcee0 2024-01-23T08:50:00.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.044+0000 7f323a7fc700 1 -- 172.21.15.59:0/2394864545 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3234013660 con 0x7f32440fcee0 2024-01-23T08:50:00.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.044+0000 7f324bb53700 1 -- 172.21.15.59:0/2394864545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f324410e190 con 0x7f32440fcee0 2024-01-23T08:50:00.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.045+0000 7f324bb53700 1 -- 172.21.15.59:0/2394864545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f324404bd60 con 0x7f32440fcee0 2024-01-23T08:50:00.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.046+0000 7f323a7fc700 1 -- 172.21.15.59:0/2394864545 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f323401ea60 con 0x7f32440fcee0 2024-01-23T08:50:00.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.046+0000 7f323a7fc700 1 --2- 172.21.15.59:0/2394864545 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f3230069030 0x7f323006b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:00.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.046+0000 7f323a7fc700 1 -- 172.21.15.59:0/2394864545 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f323408aba0 con 0x7f32440fcee0 2024-01-23T08:50:00.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.046+0000 7f32498ef700 1 --2- 172.21.15.59:0/2394864545 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f3230069030 0x7f323006b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:00.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.047+0000 7f32498ef700 1 --2- 172.21.15.59:0/2394864545 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f3230069030 0x7f323006b4e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f32440fcd40 tx=0x7f324000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:00.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.052+0000 7f323a7fc700 1 -- 172.21.15.59:0/2394864545 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f323405aba0 con 0x7f32440fcee0 2024-01-23T08:50:00.211 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:49:59.999+0000 7f5c9756c700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is offline; 1 filesystem is online with fewer MDS than max_mds 2024-01-23T08:50:00.211 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:49:59.999+0000 7f5c9756c700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-01-23T08:50:00.211 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:49:59.999+0000 7f5c9756c700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-01-23T08:50:00.211 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:49:59.999+0000 7f5c9756c700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_UP_LESS_THAN_MAX: 1 filesystem is online with fewer MDS than max_mds 2024-01-23T08:50:00.212 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:49:59.999+0000 7f5c9756c700 -1 log_channel(cluster) log [ERR] : fs cephfs has 0 MDS online, but wants 1 2024-01-23T08:50:00.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.297+0000 7f324bb53700 1 -- 172.21.15.59:0/2394864545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f324404bd60 con 0x7f32440fcee0 2024-01-23T08:50:00.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.297+0000 7f323a7fc700 1 -- 172.21.15.59:0/2394864545 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1076 (secure 0 0 0) 0x7f323405a730 con 0x7f32440fcee0 2024-01-23T08:50:00.294 INFO:teuthology.orchestra.run.smithi059.stdout:e2 2024-01-23T08:50:00.294 INFO:teuthology.orchestra.run.smithi059.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-01-23T08:50:00.295 INFO:teuthology.orchestra.run.smithi059.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-23T08:50:00.295 INFO:teuthology.orchestra.run.smithi059.stdout:legacy client fscid: 1 2024-01-23T08:50:00.295 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:50:00.295 INFO:teuthology.orchestra.run.smithi059.stdout:Filesystem 'cephfs' (1) 2024-01-23T08:50:00.295 INFO:teuthology.orchestra.run.smithi059.stdout:fs_name cephfs 2024-01-23T08:50:00.295 INFO:teuthology.orchestra.run.smithi059.stdout:epoch 2 2024-01-23T08:50:00.295 INFO:teuthology.orchestra.run.smithi059.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-01-23T08:50:00.295 INFO:teuthology.orchestra.run.smithi059.stdout:created 2024-01-23T08:49:56.450575+0000 2024-01-23T08:50:00.295 INFO:teuthology.orchestra.run.smithi059.stdout:modified 2024-01-23T08:49:56.450643+0000 2024-01-23T08:50:00.295 INFO:teuthology.orchestra.run.smithi059.stdout:tableserver 0 2024-01-23T08:50:00.295 INFO:teuthology.orchestra.run.smithi059.stdout:root 0 2024-01-23T08:50:00.295 INFO:teuthology.orchestra.run.smithi059.stdout:session_timeout 60 2024-01-23T08:50:00.295 INFO:teuthology.orchestra.run.smithi059.stdout:session_autoclose 300 2024-01-23T08:50:00.295 INFO:teuthology.orchestra.run.smithi059.stdout:max_file_size 1099511627776 2024-01-23T08:50:00.296 INFO:teuthology.orchestra.run.smithi059.stdout:required_client_features {} 2024-01-23T08:50:00.296 INFO:teuthology.orchestra.run.smithi059.stdout:last_failure 0 2024-01-23T08:50:00.296 INFO:teuthology.orchestra.run.smithi059.stdout:last_failure_osd_epoch 0 2024-01-23T08:50:00.296 INFO:teuthology.orchestra.run.smithi059.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-23T08:50:00.296 INFO:teuthology.orchestra.run.smithi059.stdout:max_mds 1 2024-01-23T08:50:00.296 INFO:teuthology.orchestra.run.smithi059.stdout:in 2024-01-23T08:50:00.296 INFO:teuthology.orchestra.run.smithi059.stdout:up {} 2024-01-23T08:50:00.296 INFO:teuthology.orchestra.run.smithi059.stdout:failed 2024-01-23T08:50:00.296 INFO:teuthology.orchestra.run.smithi059.stdout:damaged 2024-01-23T08:50:00.296 INFO:teuthology.orchestra.run.smithi059.stdout:stopped 2024-01-23T08:50:00.296 INFO:teuthology.orchestra.run.smithi059.stdout:data_pools [3] 2024-01-23T08:50:00.296 INFO:teuthology.orchestra.run.smithi059.stdout:metadata_pool 2 2024-01-23T08:50:00.296 INFO:teuthology.orchestra.run.smithi059.stdout:inline_data disabled 2024-01-23T08:50:00.297 INFO:teuthology.orchestra.run.smithi059.stdout:balancer 2024-01-23T08:50:00.297 INFO:teuthology.orchestra.run.smithi059.stdout:standby_count_wanted 0 2024-01-23T08:50:00.297 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:50:00.297 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:50:00.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.299+0000 7f324bb53700 1 -- 172.21.15.59:0/2394864545 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f3230069030 msgr2=0x7f323006b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:00.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.299+0000 7f324bb53700 1 --2- 172.21.15.59:0/2394864545 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f3230069030 0x7f323006b4e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f32440fcd40 tx=0x7f324000b040 comp rx=0 tx=0).stop 2024-01-23T08:50:00.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.299+0000 7f324bb53700 1 -- 172.21.15.59:0/2394864545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32440fcee0 msgr2=0x7f324410cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:00.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.299+0000 7f324bb53700 1 --2- 172.21.15.59:0/2394864545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32440fcee0 0x7f324410cb90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f32340093b0 tx=0x7f3234009e30 comp rx=0 tx=0).stop 2024-01-23T08:50:00.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.299+0000 7f324bb53700 1 -- 172.21.15.59:0/2394864545 shutdown_connections 2024-01-23T08:50:00.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.299+0000 7f324bb53700 1 --2- 172.21.15.59:0/2394864545 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f3230069030 0x7f323006b4e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:00.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.299+0000 7f324bb53700 1 --2- 172.21.15.59:0/2394864545 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32440fbcd0 0x7f324410c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:00.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.299+0000 7f324bb53700 1 --2- 172.21.15.59:0/2394864545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32440fcee0 0x7f324410cb90 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:00.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.299+0000 7f324bb53700 1 -- 172.21.15.59:0/2394864545 >> 172.21.15.59:0/2394864545 conn(0x7f32440f7370 msgr2=0x7f3244100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:00.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.299+0000 7f324bb53700 1 -- 172.21.15.59:0/2394864545 shutdown_connections 2024-01-23T08:50:00.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:00.299+0000 7f324bb53700 1 -- 172.21.15.59:0/2394864545 wait complete. 2024-01-23T08:50:00.298 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 2 2024-01-23T08:50:00.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:00 smithi059 ceph-mon[28424]: Health detail: HEALTH_ERR 1 filesystem is offline; 1 filesystem is online with fewer MDS than max_mds 2024-01-23T08:50:00.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:00 smithi059 ceph-mon[28424]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-01-23T08:50:00.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:00 smithi059 ceph-mon[28424]: fs cephfs is offline because no MDS is active for it. 2024-01-23T08:50:00.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:00 smithi059 ceph-mon[28424]: [WRN] MDS_UP_LESS_THAN_MAX: 1 filesystem is online with fewer MDS than max_mds 2024-01-23T08:50:00.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:00 smithi059 ceph-mon[28424]: fs cephfs has 0 MDS online, but wants 1 2024-01-23T08:50:00.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:00 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/2394864545' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-01-23T08:50:00.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:00 smithi137 ceph-mon[35787]: Health detail: HEALTH_ERR 1 filesystem is offline; 1 filesystem is online with fewer MDS than max_mds 2024-01-23T08:50:00.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:00 smithi137 ceph-mon[35787]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-01-23T08:50:00.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:00 smithi137 ceph-mon[35787]: fs cephfs is offline because no MDS is active for it. 2024-01-23T08:50:00.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:00 smithi137 ceph-mon[35787]: [WRN] MDS_UP_LESS_THAN_MAX: 1 filesystem is online with fewer MDS than max_mds 2024-01-23T08:50:00.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:00 smithi137 ceph-mon[35787]: fs cephfs has 0 MDS online, but wants 1 2024-01-23T08:50:00.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:00 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/2394864545' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-01-23T08:50:01.055 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-01-23T08:50:01.065 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2024-01-23T08:50:01.066 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-01-23T08:50:01.392 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:50:01.623 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:01 smithi059 ceph-mon[28424]: pgmap v179: 65 pgs: 16 creating+peering, 23 unknown, 26 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:50:01.623 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:01 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:01.623 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:01 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:01.623 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:01 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:01.623 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:01 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi137.mgacmp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-23T08:50:01.623 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:01 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi137.mgacmp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-23T08:50:01.624 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:01 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:50:01.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:01 smithi137 ceph-mon[35787]: pgmap v179: 65 pgs: 16 creating+peering, 23 unknown, 26 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:50:01.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:01 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:01.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:01 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:01.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:01 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:01.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:01 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi137.mgacmp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-23T08:50:01.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:01 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi137.mgacmp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-23T08:50:01.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:01 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:50:02.506 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:02 smithi059 ceph-mon[28424]: Deploying daemon mds.cephfs.smithi137.mgacmp on smithi137 2024-01-23T08:50:02.506 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:02 smithi059 ceph-mon[28424]: mds.? [v2:172.21.15.59:6826/3125523205,v1:172.21.15.59:6827/3125523205] up:boot 2024-01-23T08:50:02.506 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:02 smithi059 ceph-mon[28424]: daemon mds.cephfs.smithi059.lsxsmm assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-01-23T08:50:02.506 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:02 smithi059 ceph-mon[28424]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-01-23T08:50:02.506 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:02 smithi059 ceph-mon[28424]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-01-23T08:50:02.506 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:02 smithi059 ceph-mon[28424]: Cluster is now healthy 2024-01-23T08:50:02.506 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:02 smithi059 ceph-mon[28424]: fsmap cephfs:0 1 up:standby 2024-01-23T08:50:02.506 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:02 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi059.lsxsmm"}]: dispatch 2024-01-23T08:50:02.506 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:02 smithi059 ceph-mon[28424]: fsmap cephfs:1 {0=cephfs.smithi059.lsxsmm=up:creating} 2024-01-23T08:50:02.506 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:02 smithi059 ceph-mon[28424]: daemon mds.cephfs.smithi059.lsxsmm is now active in filesystem cephfs as rank 0 2024-01-23T08:50:02.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:02 smithi137 ceph-mon[35787]: Deploying daemon mds.cephfs.smithi137.mgacmp on smithi137 2024-01-23T08:50:02.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:02 smithi137 ceph-mon[35787]: mds.? [v2:172.21.15.59:6826/3125523205,v1:172.21.15.59:6827/3125523205] up:boot 2024-01-23T08:50:02.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:02 smithi137 ceph-mon[35787]: daemon mds.cephfs.smithi059.lsxsmm assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-01-23T08:50:02.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:02 smithi137 ceph-mon[35787]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-01-23T08:50:02.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:02 smithi137 ceph-mon[35787]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-01-23T08:50:02.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:02 smithi137 ceph-mon[35787]: Cluster is now healthy 2024-01-23T08:50:02.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:02 smithi137 ceph-mon[35787]: fsmap cephfs:0 1 up:standby 2024-01-23T08:50:02.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:02 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi059.lsxsmm"}]: dispatch 2024-01-23T08:50:02.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:02 smithi137 ceph-mon[35787]: fsmap cephfs:1 {0=cephfs.smithi059.lsxsmm=up:creating} 2024-01-23T08:50:02.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:02 smithi137 ceph-mon[35787]: daemon mds.cephfs.smithi059.lsxsmm is now active in filesystem cephfs as rank 0 2024-01-23T08:50:02.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.869+0000 7fd3ed4b6700 1 -- 172.21.15.59:0/3234062427 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3e80fcee0 msgr2=0x7fd3e80fd330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:02.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.869+0000 7fd3ed4b6700 1 --2- 172.21.15.59:0/3234062427 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3e80fcee0 0x7fd3e80fd330 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8004940 tx=0x7fd3d8009e30 comp rx=0 tx=0).stop 2024-01-23T08:50:02.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.869+0000 7fd3ed4b6700 1 -- 172.21.15.59:0/3234062427 shutdown_connections 2024-01-23T08:50:02.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.869+0000 7fd3ed4b6700 1 --2- 172.21.15.59:0/3234062427 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3e80fcee0 0x7fd3e80fd330 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:02.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.869+0000 7fd3ed4b6700 1 --2- 172.21.15.59:0/3234062427 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3e80fbcd0 0x7fd3e80fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:02.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.869+0000 7fd3ed4b6700 1 -- 172.21.15.59:0/3234062427 >> 172.21.15.59:0/3234062427 conn(0x7fd3e80f7370 msgr2=0x7fd3e80f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:02.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.869+0000 7fd3ed4b6700 1 -- 172.21.15.59:0/3234062427 shutdown_connections 2024-01-23T08:50:02.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.869+0000 7fd3ed4b6700 1 -- 172.21.15.59:0/3234062427 wait complete. 2024-01-23T08:50:02.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.870+0000 7fd3ed4b6700 1 Processor -- start 2024-01-23T08:50:02.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.870+0000 7fd3ed4b6700 1 -- start start 2024-01-23T08:50:02.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.870+0000 7fd3ed4b6700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3e80fbcd0 0x7fd3e80ffda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:02.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.870+0000 7fd3ed4b6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3e80fcee0 0x7fd3e80fe220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:02.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.870+0000 7fd3ed4b6700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e80fe820 con 0x7fd3e80fcee0 2024-01-23T08:50:02.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.870+0000 7fd3ed4b6700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e80fe990 con 0x7fd3e80fbcd0 2024-01-23T08:50:02.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.870+0000 7fd3e67fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3e80fcee0 0x7fd3e80fe220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:02.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.870+0000 7fd3e67fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3e80fcee0 0x7fd3e80fe220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56372/0 (socket says 172.21.15.59:56372) 2024-01-23T08:50:02.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.870+0000 7fd3e67fc700 1 -- 172.21.15.59:0/3412667820 learned_addr learned my addr 172.21.15.59:0/3412667820 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:50:02.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.870+0000 7fd3e6ffd700 1 --2- 172.21.15.59:0/3412667820 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3e80fbcd0 0x7fd3e80ffda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:02.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.871+0000 7fd3e67fc700 1 -- 172.21.15.59:0/3412667820 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3e80fbcd0 msgr2=0x7fd3e80ffda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:02.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.871+0000 7fd3e67fc700 1 --2- 172.21.15.59:0/3412667820 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3e80fbcd0 0x7fd3e80ffda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:02.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.871+0000 7fd3e67fc700 1 -- 172.21.15.59:0/3412667820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d8005040 con 0x7fd3e80fcee0 2024-01-23T08:50:02.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.871+0000 7fd3e6ffd700 1 --2- 172.21.15.59:0/3412667820 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3e80fbcd0 0x7fd3e80ffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-23T08:50:02.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.871+0000 7fd3e67fc700 1 --2- 172.21.15.59:0/3412667820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3e80fcee0 0x7fd3e80fe220 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8009360 tx=0x7fd3d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:02.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.871+0000 7fd3cf7fe700 1 -- 172.21.15.59:0/3412667820 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3d800e040 con 0x7fd3e80fcee0 2024-01-23T08:50:02.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.871+0000 7fd3cf7fe700 1 -- 172.21.15.59:0/3412667820 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd3d800bcd0 con 0x7fd3e80fcee0 2024-01-23T08:50:02.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.871+0000 7fd3ed4b6700 1 -- 172.21.15.59:0/3412667820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e80fec10 con 0x7fd3e80fcee0 2024-01-23T08:50:02.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.871+0000 7fd3cf7fe700 1 -- 172.21.15.59:0/3412667820 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3d801c630 con 0x7fd3e80fcee0 2024-01-23T08:50:02.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.871+0000 7fd3ed4b6700 1 -- 172.21.15.59:0/3412667820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e8115790 con 0x7fd3e80fcee0 2024-01-23T08:50:02.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.872+0000 7fd3ed4b6700 1 -- 172.21.15.59:0/3412667820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e804bd60 con 0x7fd3e80fcee0 2024-01-23T08:50:02.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.873+0000 7fd3cf7fe700 1 -- 172.21.15.59:0/3412667820 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fd3d8013960 con 0x7fd3e80fcee0 2024-01-23T08:50:02.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.873+0000 7fd3cf7fe700 1 --2- 172.21.15.59:0/3412667820 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fd3d0068f60 0x7fd3d006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:02.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.873+0000 7fd3cf7fe700 1 -- 172.21.15.59:0/3412667820 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fd3d8088110 con 0x7fd3e80fcee0 2024-01-23T08:50:02.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.873+0000 7fd3e6ffd700 1 --2- 172.21.15.59:0/3412667820 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fd3d0068f60 0x7fd3d006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:02.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.873+0000 7fd3e6ffd700 1 --2- 172.21.15.59:0/3412667820 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fd3d0068f60 0x7fd3d006b410 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc009fd0 tx=0x7fd3dc009f00 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:02.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:02.877+0000 7fd3cf7fe700 1 -- 172.21.15.59:0/3412667820 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fd3d80581c0 con 0x7fd3e80fcee0 2024-01-23T08:50:03.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:03.114+0000 7fd3ed4b6700 1 -- 172.21.15.59:0/3412667820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7fd3e8115b70 con 0x7fd3e80fcee0 2024-01-23T08:50:03.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:03 smithi059 ceph-mon[28424]: pgmap v180: 65 pgs: 6 creating+peering, 59 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:50:03.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:03 smithi059 ceph-mon[28424]: mds.? [v2:172.21.15.59:6826/3125523205,v1:172.21.15.59:6827/3125523205] up:active 2024-01-23T08:50:03.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:03 smithi059 ceph-mon[28424]: fsmap cephfs:1 {0=cephfs.smithi059.lsxsmm=up:active} 2024-01-23T08:50:03.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:03 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/3412667820' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-01-23T08:50:03.791 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:03 smithi137 ceph-mon[35787]: pgmap v180: 65 pgs: 6 creating+peering, 59 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-23T08:50:03.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:03 smithi137 ceph-mon[35787]: mds.? [v2:172.21.15.59:6826/3125523205,v1:172.21.15.59:6827/3125523205] up:active 2024-01-23T08:50:03.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:03 smithi137 ceph-mon[35787]: fsmap cephfs:1 {0=cephfs.smithi059.lsxsmm=up:active} 2024-01-23T08:50:03.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:03 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/3412667820' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-01-23T08:50:04.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:04.058+0000 7fd3cf7fe700 1 -- 172.21.15.59:0/3412667820 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7fd3d8057d50 con 0x7fd3e80fcee0 2024-01-23T08:50:04.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:04.060+0000 7fd3ed4b6700 1 -- 172.21.15.59:0/3412667820 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fd3d0068f60 msgr2=0x7fd3d006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:04.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:04.060+0000 7fd3ed4b6700 1 --2- 172.21.15.59:0/3412667820 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fd3d0068f60 0x7fd3d006b410 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc009fd0 tx=0x7fd3dc009f00 comp rx=0 tx=0).stop 2024-01-23T08:50:04.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:04.061+0000 7fd3ed4b6700 1 -- 172.21.15.59:0/3412667820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3e80fcee0 msgr2=0x7fd3e80fe220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:04.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:04.061+0000 7fd3ed4b6700 1 --2- 172.21.15.59:0/3412667820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3e80fcee0 0x7fd3e80fe220 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8009360 tx=0x7fd3d8009e30 comp rx=0 tx=0).stop 2024-01-23T08:50:04.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:04.061+0000 7fd3ed4b6700 1 -- 172.21.15.59:0/3412667820 shutdown_connections 2024-01-23T08:50:04.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:04.061+0000 7fd3ed4b6700 1 --2- 172.21.15.59:0/3412667820 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fd3d0068f60 0x7fd3d006b410 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:04.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:04.061+0000 7fd3ed4b6700 1 --2- 172.21.15.59:0/3412667820 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3e80fbcd0 0x7fd3e80ffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:04.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:04.061+0000 7fd3ed4b6700 1 --2- 172.21.15.59:0/3412667820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3e80fcee0 0x7fd3e80fe220 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:04.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:04.061+0000 7fd3ed4b6700 1 -- 172.21.15.59:0/3412667820 >> 172.21.15.59:0/3412667820 conn(0x7fd3e80f7370 msgr2=0x7fd3e8065550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:04.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:04.061+0000 7fd3ed4b6700 1 -- 172.21.15.59:0/3412667820 shutdown_connections 2024-01-23T08:50:04.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:04.061+0000 7fd3ed4b6700 1 -- 172.21.15.59:0/3412667820 wait complete. 2024-01-23T08:50:04.652 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-01-23T08:50:04.663 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2024-01-23T08:50:04.663 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-01-23T08:50:04.920 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:04 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:04.920 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:04 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:04.921 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:04 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:04.921 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:04 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi059.twpxfz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-23T08:50:04.921 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:04 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi059.twpxfz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-23T08:50:04.921 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:04 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:50:04.921 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:04 smithi059 ceph-mon[28424]: Deploying daemon mds.cephfs.smithi059.twpxfz on smithi059 2024-01-23T08:50:04.921 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:04 smithi059 ceph-mon[28424]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-01-23T08:50:04.921 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:04 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/3412667820' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-01-23T08:50:04.921 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:04 smithi059 ceph-mon[28424]: mds.? [v2:172.21.15.137:6824/1814038594,v1:172.21.15.137:6825/1814038594] up:boot 2024-01-23T08:50:04.921 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:04 smithi059 ceph-mon[28424]: daemon mds.cephfs.smithi137.mgacmp assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-01-23T08:50:04.921 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:04 smithi059 ceph-mon[28424]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-01-23T08:50:04.921 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:04 smithi059 ceph-mon[28424]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-01-23T08:50:04.921 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:04 smithi059 ceph-mon[28424]: fsmap cephfs:1 {0=cephfs.smithi059.lsxsmm=up:active} 1 up:standby 2024-01-23T08:50:04.921 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:04 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.mgacmp"}]: dispatch 2024-01-23T08:50:04.921 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:04 smithi059 ceph-mon[28424]: fsmap cephfs:2 {0=cephfs.smithi059.lsxsmm=up:active,1=cephfs.smithi137.mgacmp=up:creating} 2024-01-23T08:50:04.922 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:04 smithi059 ceph-mon[28424]: daemon mds.cephfs.smithi137.mgacmp is now active in filesystem cephfs as rank 1 2024-01-23T08:50:04.922 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:04 smithi059 ceph-mon[28424]: pgmap v181: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 907 B/s wr, 2 op/s 2024-01-23T08:50:05.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:04 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:05.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:04 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:05.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:04 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:05.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:04 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi059.twpxfz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-23T08:50:05.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:04 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi059.twpxfz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-23T08:50:05.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:04 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:50:05.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:04 smithi137 ceph-mon[35787]: Deploying daemon mds.cephfs.smithi059.twpxfz on smithi059 2024-01-23T08:50:05.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:04 smithi137 ceph-mon[35787]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-01-23T08:50:05.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:04 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/3412667820' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-01-23T08:50:05.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:04 smithi137 ceph-mon[35787]: mds.? [v2:172.21.15.137:6824/1814038594,v1:172.21.15.137:6825/1814038594] up:boot 2024-01-23T08:50:05.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:04 smithi137 ceph-mon[35787]: daemon mds.cephfs.smithi137.mgacmp assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-01-23T08:50:05.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:04 smithi137 ceph-mon[35787]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-01-23T08:50:05.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:04 smithi137 ceph-mon[35787]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-01-23T08:50:05.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:04 smithi137 ceph-mon[35787]: fsmap cephfs:1 {0=cephfs.smithi059.lsxsmm=up:active} 1 up:standby 2024-01-23T08:50:05.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:04 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.mgacmp"}]: dispatch 2024-01-23T08:50:05.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:04 smithi137 ceph-mon[35787]: fsmap cephfs:2 {0=cephfs.smithi059.lsxsmm=up:active,1=cephfs.smithi137.mgacmp=up:creating} 2024-01-23T08:50:05.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:04 smithi137 ceph-mon[35787]: daemon mds.cephfs.smithi137.mgacmp is now active in filesystem cephfs as rank 1 2024-01-23T08:50:05.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:04 smithi137 ceph-mon[35787]: pgmap v181: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 907 B/s wr, 2 op/s 2024-01-23T08:50:05.731 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:50:06.103 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:06 smithi137 ceph-mon[35787]: mds.? [v2:172.21.15.137:6824/1814038594,v1:172.21.15.137:6825/1814038594] up:active 2024-01-23T08:50:06.103 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:06 smithi137 ceph-mon[35787]: fsmap cephfs:2 {0=cephfs.smithi059.lsxsmm=up:active,1=cephfs.smithi137.mgacmp=up:active} 2024-01-23T08:50:06.193 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:06 smithi059 ceph-mon[28424]: mds.? [v2:172.21.15.137:6824/1814038594,v1:172.21.15.137:6825/1814038594] up:active 2024-01-23T08:50:06.194 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:06 smithi059 ceph-mon[28424]: fsmap cephfs:2 {0=cephfs.smithi059.lsxsmm=up:active,1=cephfs.smithi137.mgacmp=up:active} 2024-01-23T08:50:07.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:07 smithi059 ceph-mon[28424]: pgmap v182: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s wr, 7 op/s 2024-01-23T08:50:07.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:07 smithi137 ceph-mon[35787]: pgmap v182: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s wr, 7 op/s 2024-01-23T08:50:07.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.353+0000 7f9e7e72d700 1 -- 172.21.15.59:0/748298035 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e780fcdd0 msgr2=0x7f9e780fd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:07.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.353+0000 7f9e7e72d700 1 --2- 172.21.15.59:0/748298035 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e780fcdd0 0x7f9e780fd1e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9e68004940 tx=0x7f9e68009e30 comp rx=0 tx=0).stop 2024-01-23T08:50:07.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.353+0000 7f9e7e72d700 1 -- 172.21.15.59:0/748298035 shutdown_connections 2024-01-23T08:50:07.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.353+0000 7f9e7e72d700 1 --2- 172.21.15.59:0/748298035 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e780fdfe0 0x7f9e780fe430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:07.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.353+0000 7f9e7e72d700 1 --2- 172.21.15.59:0/748298035 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e780fcdd0 0x7f9e780fd1e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:07.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.353+0000 7f9e7e72d700 1 -- 172.21.15.59:0/748298035 >> 172.21.15.59:0/748298035 conn(0x7f9e780f84b0 msgr2=0x7f9e780fa8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:07.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.353+0000 7f9e7e72d700 1 -- 172.21.15.59:0/748298035 shutdown_connections 2024-01-23T08:50:07.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.353+0000 7f9e7e72d700 1 -- 172.21.15.59:0/748298035 wait complete. 2024-01-23T08:50:07.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.354+0000 7f9e7e72d700 1 Processor -- start 2024-01-23T08:50:07.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.354+0000 7f9e7e72d700 1 -- start start 2024-01-23T08:50:07.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.354+0000 7f9e7e72d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e780fcdd0 0x7f9e7810d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:07.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.354+0000 7f9e7e72d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e780fdfe0 0x7f9e7810dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:07.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.354+0000 7f9e7e72d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7810e320 con 0x7f9e780fdfe0 2024-01-23T08:50:07.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.354+0000 7f9e7e72d700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7810ebc0 con 0x7f9e780fcdd0 2024-01-23T08:50:07.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.355+0000 7f9e777fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e780fdfe0 0x7f9e7810dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:07.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.355+0000 7f9e777fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e780fdfe0 0x7f9e7810dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56384/0 (socket says 172.21.15.59:56384) 2024-01-23T08:50:07.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.355+0000 7f9e77fff700 1 --2- 172.21.15.59:0/2744552897 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e780fcdd0 0x7f9e7810d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:07.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.355+0000 7f9e777fe700 1 -- 172.21.15.59:0/2744552897 learned_addr learned my addr 172.21.15.59:0/2744552897 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:50:07.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.355+0000 7f9e77fff700 1 -- 172.21.15.59:0/2744552897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e780fdfe0 msgr2=0x7f9e7810dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:07.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.355+0000 7f9e77fff700 1 --2- 172.21.15.59:0/2744552897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e780fdfe0 0x7f9e7810dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:07.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.355+0000 7f9e77fff700 1 -- 172.21.15.59:0/2744552897 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e68005040 con 0x7f9e780fcdd0 2024-01-23T08:50:07.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.355+0000 7f9e77fff700 1 --2- 172.21.15.59:0/2744552897 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e780fcdd0 0x7f9e7810d7e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9e68009fd0 tx=0x7f9e68009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:07.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.355+0000 7f9e74ff9700 1 -- 172.21.15.59:0/2744552897 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e6800e040 con 0x7f9e780fcdd0 2024-01-23T08:50:07.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.355+0000 7f9e7e72d700 1 -- 172.21.15.59:0/2744552897 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e7810ee40 con 0x7f9e780fcdd0 2024-01-23T08:50:07.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.355+0000 7f9e7e72d700 1 -- 172.21.15.59:0/2744552897 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e7810f380 con 0x7f9e780fcdd0 2024-01-23T08:50:07.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.355+0000 7f9e74ff9700 1 -- 172.21.15.59:0/2744552897 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e6800bc50 con 0x7f9e780fcdd0 2024-01-23T08:50:07.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.355+0000 7f9e74ff9700 1 -- 172.21.15.59:0/2744552897 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e68025640 con 0x7f9e780fcdd0 2024-01-23T08:50:07.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.357+0000 7f9e7e72d700 1 -- 172.21.15.59:0/2744552897 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e781076e0 con 0x7f9e780fcdd0 2024-01-23T08:50:07.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.357+0000 7f9e74ff9700 1 -- 172.21.15.59:0/2744552897 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f9e6801c3f0 con 0x7f9e780fcdd0 2024-01-23T08:50:07.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.357+0000 7f9e74ff9700 1 --2- 172.21.15.59:0/2744552897 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f9e60068f60 0x7f9e6006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:07.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.357+0000 7f9e777fe700 1 --2- 172.21.15.59:0/2744552897 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f9e60068f60 0x7f9e6006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:07.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.357+0000 7f9e74ff9700 1 -- 172.21.15.59:0/2744552897 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f9e68087ff0 con 0x7f9e780fcdd0 2024-01-23T08:50:07.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.357+0000 7f9e777fe700 1 --2- 172.21.15.59:0/2744552897 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f9e60068f60 0x7f9e6006b410 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c0041d0 tx=0x7f9e6c00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:07.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.362+0000 7f9e74ff9700 1 -- 172.21.15.59:0/2744552897 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f9e68057fc0 con 0x7f9e780fcdd0 2024-01-23T08:50:07.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:07.614+0000 7f9e7e72d700 1 -- 172.21.15.59:0/2744552897 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f9e7804bd60 con 0x7f9e780fcdd0 2024-01-23T08:50:08.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:08.077+0000 7f9e74ff9700 1 -- 172.21.15.59:0/2744552897 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v9) v1 ==== 121+0+0 (secure 0 0 0) 0x7f9e68057b50 con 0x7f9e780fcdd0 2024-01-23T08:50:08.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:08.080+0000 7f9e7e72d700 1 -- 172.21.15.59:0/2744552897 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f9e60068f60 msgr2=0x7f9e6006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:08.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:08.080+0000 7f9e7e72d700 1 --2- 172.21.15.59:0/2744552897 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f9e60068f60 0x7f9e6006b410 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c0041d0 tx=0x7f9e6c00b040 comp rx=0 tx=0).stop 2024-01-23T08:50:08.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:08.080+0000 7f9e7e72d700 1 -- 172.21.15.59:0/2744552897 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e780fcdd0 msgr2=0x7f9e7810d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:08.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:08.080+0000 7f9e7e72d700 1 --2- 172.21.15.59:0/2744552897 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e780fcdd0 0x7f9e7810d7e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9e68009fd0 tx=0x7f9e68009e30 comp rx=0 tx=0).stop 2024-01-23T08:50:08.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:08.080+0000 7f9e7e72d700 1 -- 172.21.15.59:0/2744552897 shutdown_connections 2024-01-23T08:50:08.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:08.080+0000 7f9e7e72d700 1 --2- 172.21.15.59:0/2744552897 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f9e60068f60 0x7f9e6006b410 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:08.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:08.080+0000 7f9e7e72d700 1 --2- 172.21.15.59:0/2744552897 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e780fcdd0 0x7f9e7810d7e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:08.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:08.080+0000 7f9e7e72d700 1 --2- 172.21.15.59:0/2744552897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e780fdfe0 0x7f9e7810dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:08.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:08.080+0000 7f9e7e72d700 1 -- 172.21.15.59:0/2744552897 >> 172.21.15.59:0/2744552897 conn(0x7f9e780f84b0 msgr2=0x7f9e780fa880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:08.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:08.081+0000 7f9e7e72d700 1 -- 172.21.15.59:0/2744552897 shutdown_connections 2024-01-23T08:50:08.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:08.081+0000 7f9e7e72d700 1 -- 172.21.15.59:0/2744552897 wait complete. 2024-01-23T08:50:08.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:08 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/2744552897' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-01-23T08:50:08.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:08 smithi059 ceph-mon[28424]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-01-23T08:50:08.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:08 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:08.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:08 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:08.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:08 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:08.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:08 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi137.scidwx", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-23T08:50:08.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:08 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi137.scidwx", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-23T08:50:08.222 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:08 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:50:08.441 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:08 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/2744552897' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-01-23T08:50:08.442 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:08 smithi137 ceph-mon[35787]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-01-23T08:50:08.442 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:08 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:08.442 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:08 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:08.442 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:08 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:08.442 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:08 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi137.scidwx", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-23T08:50:08.442 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:08 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi137.scidwx", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-23T08:50:08.442 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:08 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:50:08.692 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-01-23T08:50:08.702 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2024-01-23T08:50:08.702 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-01-23T08:50:08.965 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:50:09.327 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:09 smithi059 ceph-mon[28424]: Deploying daemon mds.cephfs.smithi137.scidwx on smithi137 2024-01-23T08:50:09.327 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:09 smithi059 ceph-mon[28424]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-01-23T08:50:09.328 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:09 smithi059 ceph-mon[28424]: Cluster is now healthy 2024-01-23T08:50:09.328 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:09 smithi059 ceph-mon[28424]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-01-23T08:50:09.328 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:09 smithi059 ceph-mon[28424]: mds.? [v2:172.21.15.59:6828/4118577293,v1:172.21.15.59:6829/4118577293] up:boot 2024-01-23T08:50:09.328 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:09 smithi059 ceph-mon[28424]: fsmap cephfs:2 {0=cephfs.smithi059.lsxsmm=up:active,1=cephfs.smithi137.mgacmp=up:active} 1 up:standby 2024-01-23T08:50:09.328 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:09 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi059.twpxfz"}]: dispatch 2024-01-23T08:50:09.328 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:09 smithi059 ceph-mon[28424]: fsmap cephfs:2 {0=cephfs.smithi059.lsxsmm=up:active,1=cephfs.smithi137.mgacmp=up:active} 1 up:standby-replay 2024-01-23T08:50:09.328 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:09 smithi059 ceph-mon[28424]: pgmap v183: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s wr, 8 op/s 2024-01-23T08:50:09.456 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:09 smithi137 ceph-mon[35787]: Deploying daemon mds.cephfs.smithi137.scidwx on smithi137 2024-01-23T08:50:09.456 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:09 smithi137 ceph-mon[35787]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-01-23T08:50:09.456 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:09 smithi137 ceph-mon[35787]: Cluster is now healthy 2024-01-23T08:50:09.456 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:09 smithi137 ceph-mon[35787]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-01-23T08:50:09.456 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:09 smithi137 ceph-mon[35787]: mds.? [v2:172.21.15.59:6828/4118577293,v1:172.21.15.59:6829/4118577293] up:boot 2024-01-23T08:50:09.456 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:09 smithi137 ceph-mon[35787]: fsmap cephfs:2 {0=cephfs.smithi059.lsxsmm=up:active,1=cephfs.smithi137.mgacmp=up:active} 1 up:standby 2024-01-23T08:50:09.457 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:09 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi059.twpxfz"}]: dispatch 2024-01-23T08:50:09.457 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:09 smithi137 ceph-mon[35787]: fsmap cephfs:2 {0=cephfs.smithi059.lsxsmm=up:active,1=cephfs.smithi137.mgacmp=up:active} 1 up:standby-replay 2024-01-23T08:50:09.457 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:09 smithi137 ceph-mon[35787]: pgmap v183: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s wr, 8 op/s 2024-01-23T08:50:10.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.651+0000 7fcacfa3b700 1 -- 172.21.15.59:0/575017959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcac80fd910 msgr2=0x7fcac80ffcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:10.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.651+0000 7fcacfa3b700 1 --2- 172.21.15.59:0/575017959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcac80fd910 0x7fcac80ffcf0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fcac4004940 tx=0x7fcac4009e30 comp rx=0 tx=0).stop 2024-01-23T08:50:10.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.651+0000 7fcacfa3b700 1 -- 172.21.15.59:0/575017959 shutdown_connections 2024-01-23T08:50:10.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.651+0000 7fcacfa3b700 1 --2- 172.21.15.59:0/575017959 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcac8100230 0x7fcac8102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:10.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.651+0000 7fcacfa3b700 1 --2- 172.21.15.59:0/575017959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcac80fd910 0x7fcac80ffcf0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:10.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.651+0000 7fcacfa3b700 1 -- 172.21.15.59:0/575017959 >> 172.21.15.59:0/575017959 conn(0x7fcac80f7370 msgr2=0x7fcac80f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:10.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.651+0000 7fcacfa3b700 1 -- 172.21.15.59:0/575017959 shutdown_connections 2024-01-23T08:50:10.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.651+0000 7fcacfa3b700 1 -- 172.21.15.59:0/575017959 wait complete. 2024-01-23T08:50:10.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.652+0000 7fcacfa3b700 1 Processor -- start 2024-01-23T08:50:10.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.652+0000 7fcacfa3b700 1 -- start start 2024-01-23T08:50:10.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.652+0000 7fcacfa3b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcac80fd910 0x7fcac810f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:10.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.652+0000 7fcacfa3b700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcac8100230 0x7fcac810fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:10.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.652+0000 7fcacfa3b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac81104b0 con 0x7fcac80fd910 2024-01-23T08:50:10.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.652+0000 7fcacfa3b700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac8110cf0 con 0x7fcac8100230 2024-01-23T08:50:10.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.652+0000 7fcacd7d7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcac80fd910 0x7fcac810f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:10.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.652+0000 7fcacd7d7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcac80fd910 0x7fcac810f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55334/0 (socket says 172.21.15.59:55334) 2024-01-23T08:50:10.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.652+0000 7fcacd7d7700 1 -- 172.21.15.59:0/3054838626 learned_addr learned my addr 172.21.15.59:0/3054838626 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:50:10.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.652+0000 7fcaccfd6700 1 --2- 172.21.15.59:0/3054838626 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcac8100230 0x7fcac810fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:10.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.652+0000 7fcacd7d7700 1 -- 172.21.15.59:0/3054838626 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcac8100230 msgr2=0x7fcac810fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:10.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.652+0000 7fcacd7d7700 1 --2- 172.21.15.59:0/3054838626 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcac8100230 0x7fcac810fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:10.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.652+0000 7fcacd7d7700 1 -- 172.21.15.59:0/3054838626 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac4005040 con 0x7fcac80fd910 2024-01-23T08:50:10.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.652+0000 7fcaccfd6700 1 --2- 172.21.15.59:0/3054838626 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcac8100230 0x7fcac810fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-23T08:50:10.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.653+0000 7fcacd7d7700 1 --2- 172.21.15.59:0/3054838626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcac80fd910 0x7fcac810f8e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcac4009360 tx=0x7fcac4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:10.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.653+0000 7fcab9ffb700 1 -- 172.21.15.59:0/3054838626 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcac400e040 con 0x7fcac80fd910 2024-01-23T08:50:10.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.653+0000 7fcacfa3b700 1 -- 172.21.15.59:0/3054838626 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac8110f70 con 0x7fcac80fd910 2024-01-23T08:50:10.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.653+0000 7fcab9ffb700 1 -- 172.21.15.59:0/3054838626 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcac400bcd0 con 0x7fcac80fd910 2024-01-23T08:50:10.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.653+0000 7fcab9ffb700 1 -- 172.21.15.59:0/3054838626 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcac401c630 con 0x7fcac80fd910 2024-01-23T08:50:10.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.653+0000 7fcacfa3b700 1 -- 172.21.15.59:0/3054838626 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac81114b0 con 0x7fcac80fd910 2024-01-23T08:50:10.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.654+0000 7fcacfa3b700 1 -- 172.21.15.59:0/3054838626 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac804bd60 con 0x7fcac80fd910 2024-01-23T08:50:10.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.655+0000 7fcab9ffb700 1 -- 172.21.15.59:0/3054838626 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fcac4013960 con 0x7fcac80fd910 2024-01-23T08:50:10.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.655+0000 7fcab9ffb700 1 --2- 172.21.15.59:0/3054838626 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fcab4068f60 0x7fcab406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:10.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.656+0000 7fcab9ffb700 1 -- 172.21.15.59:0/3054838626 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fcac4088260 con 0x7fcac80fd910 2024-01-23T08:50:10.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.656+0000 7fcaccfd6700 1 --2- 172.21.15.59:0/3054838626 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fcab4068f60 0x7fcab406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:10.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.656+0000 7fcaccfd6700 1 --2- 172.21.15.59:0/3054838626 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fcab4068f60 0x7fcab406b410 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fcabc009b10 tx=0x7fcabc008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:10.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.660+0000 7fcab9ffb700 1 -- 172.21.15.59:0/3054838626 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fcac40581c0 con 0x7fcac80fd910 2024-01-23T08:50:10.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:10.897+0000 7fcacfa3b700 1 -- 172.21.15.59:0/3054838626 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcac8111910 con 0x7fcac80fd910 2024-01-23T08:50:11.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:11.055+0000 7fcab9ffb700 1 -- 172.21.15.59:0/3054838626 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v11) v1 ==== 168+0+0 (secure 0 0 0) 0x7fcac4057d50 con 0x7fcac80fd910 2024-01-23T08:50:11.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:11.057+0000 7fcacfa3b700 1 -- 172.21.15.59:0/3054838626 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fcab4068f60 msgr2=0x7fcab406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:11.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:11.057+0000 7fcacfa3b700 1 --2- 172.21.15.59:0/3054838626 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fcab4068f60 0x7fcab406b410 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fcabc009b10 tx=0x7fcabc008040 comp rx=0 tx=0).stop 2024-01-23T08:50:11.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:11.057+0000 7fcacfa3b700 1 -- 172.21.15.59:0/3054838626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcac80fd910 msgr2=0x7fcac810f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:11.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:11.057+0000 7fcacfa3b700 1 --2- 172.21.15.59:0/3054838626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcac80fd910 0x7fcac810f8e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcac4009360 tx=0x7fcac4009e30 comp rx=0 tx=0).stop 2024-01-23T08:50:11.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:11.057+0000 7fcacfa3b700 1 -- 172.21.15.59:0/3054838626 shutdown_connections 2024-01-23T08:50:11.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:11.057+0000 7fcacfa3b700 1 --2- 172.21.15.59:0/3054838626 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fcab4068f60 0x7fcab406b410 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:11.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:11.057+0000 7fcacfa3b700 1 --2- 172.21.15.59:0/3054838626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcac80fd910 0x7fcac810f8e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:11.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:11.057+0000 7fcacfa3b700 1 --2- 172.21.15.59:0/3054838626 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcac8100230 0x7fcac810fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:11.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:11.057+0000 7fcacfa3b700 1 -- 172.21.15.59:0/3054838626 >> 172.21.15.59:0/3054838626 conn(0x7fcac80f7370 msgr2=0x7fcac80fb700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:11.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:11.058+0000 7fcacfa3b700 1 -- 172.21.15.59:0/3054838626 shutdown_connections 2024-01-23T08:50:11.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:11.058+0000 7fcacfa3b700 1 -- 172.21.15.59:0/3054838626 wait complete. 2024-01-23T08:50:11.056 INFO:teuthology.orchestra.run.smithi059.stderr:inline data enabled 2024-01-23T08:50:11.196 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:11 smithi059 ceph-mon[28424]: pgmap v184: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s wr, 6 op/s 2024-01-23T08:50:11.196 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:11 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:50:11.196 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:11 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:11.196 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:11 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:11.196 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:11 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:11.196 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:11 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:11.196 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:11 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:11.196 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:11 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:11.197 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:11 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:50:11.197 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:11 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/3054838626' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-01-23T08:50:11.324 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:11 smithi137 ceph-mon[35787]: pgmap v184: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s wr, 6 op/s 2024-01-23T08:50:11.324 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:11 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:50:11.324 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:11 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:11.324 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:11 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:11.324 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:11 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:11.324 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:11 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:11.324 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:11 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:11.325 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:11 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:11.325 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:11 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:50:11.325 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:11 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/3054838626' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-01-23T08:50:11.864 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-01-23T08:50:11.877 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2024-01-23T08:50:11.878 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- bash -c 'ceph fs dump' 2024-01-23T08:50:12.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:12 smithi059 ceph-mon[28424]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-01-23T08:50:12.093 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:12 smithi059 ceph-mon[28424]: mds.? [v2:172.21.15.137:6826/839018117,v1:172.21.15.137:6827/839018117] up:boot 2024-01-23T08:50:12.093 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:12 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/3054838626' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-01-23T08:50:12.093 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:12 smithi059 ceph-mon[28424]: fsmap cephfs:2 {0=cephfs.smithi059.lsxsmm=up:active,1=cephfs.smithi137.mgacmp=up:active} 1 up:standby-replay 1 up:standby 2024-01-23T08:50:12.093 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:12 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.scidwx"}]: dispatch 2024-01-23T08:50:12.093 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:12 smithi059 ceph-mon[28424]: fsmap cephfs:2 {0=cephfs.smithi059.lsxsmm=up:active,1=cephfs.smithi137.mgacmp=up:active} 2 up:standby-replay 2024-01-23T08:50:12.266 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:50:12.292 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:12 smithi137 ceph-mon[35787]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-01-23T08:50:12.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:12 smithi137 ceph-mon[35787]: mds.? [v2:172.21.15.137:6826/839018117,v1:172.21.15.137:6827/839018117] up:boot 2024-01-23T08:50:12.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:12 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/3054838626' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-01-23T08:50:12.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:12 smithi137 ceph-mon[35787]: fsmap cephfs:2 {0=cephfs.smithi059.lsxsmm=up:active,1=cephfs.smithi137.mgacmp=up:active} 1 up:standby-replay 1 up:standby 2024-01-23T08:50:12.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:12 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.scidwx"}]: dispatch 2024-01-23T08:50:12.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:12 smithi137 ceph-mon[35787]: fsmap cephfs:2 {0=cephfs.smithi059.lsxsmm=up:active,1=cephfs.smithi137.mgacmp=up:active} 2 up:standby-replay 2024-01-23T08:50:13.422 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:13 smithi059 ceph-mon[28424]: mds.? [v2:172.21.15.137:6824/1814038594,v1:172.21.15.137:6825/1814038594] up:active 2024-01-23T08:50:13.422 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:13 smithi059 ceph-mon[28424]: mds.? [v2:172.21.15.59:6828/4118577293,v1:172.21.15.59:6829/4118577293] up:standby-replay 2024-01-23T08:50:13.422 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:13 smithi059 ceph-mon[28424]: Dropping low affinity active daemon mds.cephfs.smithi059.lsxsmm in favor of higher affinity standby. 2024-01-23T08:50:13.422 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:13 smithi059 ceph-mon[28424]: Replacing daemon mds.cephfs.smithi059.lsxsmm as rank 0 with standby daemon mds.cephfs.smithi059.twpxfz 2024-01-23T08:50:13.422 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:13 smithi059 ceph-mon[28424]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-01-23T08:50:13.422 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:13 smithi059 ceph-mon[28424]: fsmap cephfs:2 {0=cephfs.smithi059.lsxsmm=up:active,1=cephfs.smithi137.mgacmp=up:active} 2 up:standby-replay 2024-01-23T08:50:13.422 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:13 smithi059 ceph-mon[28424]: osdmap e47: 6 total, 6 up, 6 in 2024-01-23T08:50:13.422 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:13 smithi059 ceph-mon[28424]: fsmap cephfs:2/2 {0=cephfs.smithi059.twpxfz=up:replay,1=cephfs.smithi137.mgacmp=up:active} 1 up:standby-replay 2024-01-23T08:50:13.422 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:13 smithi059 ceph-mon[28424]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 921 B/s rd, 2.4 KiB/s wr, 9 op/s 2024-01-23T08:50:13.422 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:13 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:13.422 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:13 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:13.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:13 smithi137 ceph-mon[35787]: mds.? [v2:172.21.15.137:6824/1814038594,v1:172.21.15.137:6825/1814038594] up:active 2024-01-23T08:50:13.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:13 smithi137 ceph-mon[35787]: mds.? [v2:172.21.15.59:6828/4118577293,v1:172.21.15.59:6829/4118577293] up:standby-replay 2024-01-23T08:50:13.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:13 smithi137 ceph-mon[35787]: Dropping low affinity active daemon mds.cephfs.smithi059.lsxsmm in favor of higher affinity standby. 2024-01-23T08:50:13.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:13 smithi137 ceph-mon[35787]: Replacing daemon mds.cephfs.smithi059.lsxsmm as rank 0 with standby daemon mds.cephfs.smithi059.twpxfz 2024-01-23T08:50:13.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:13 smithi137 ceph-mon[35787]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-01-23T08:50:13.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:13 smithi137 ceph-mon[35787]: fsmap cephfs:2 {0=cephfs.smithi059.lsxsmm=up:active,1=cephfs.smithi137.mgacmp=up:active} 2 up:standby-replay 2024-01-23T08:50:13.544 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:13 smithi137 ceph-mon[35787]: osdmap e47: 6 total, 6 up, 6 in 2024-01-23T08:50:13.544 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:13 smithi137 ceph-mon[35787]: fsmap cephfs:2/2 {0=cephfs.smithi059.twpxfz=up:replay,1=cephfs.smithi137.mgacmp=up:active} 1 up:standby-replay 2024-01-23T08:50:13.544 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:13 smithi137 ceph-mon[35787]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 921 B/s rd, 2.4 KiB/s wr, 9 op/s 2024-01-23T08:50:13.544 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:13 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:13.544 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:13 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:14.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.288+0000 7f08356dd700 1 -- 172.21.15.59:0/847987273 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0830100230 msgr2=0x7f0830102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:14.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.288+0000 7f08356dd700 1 --2- 172.21.15.59:0/847987273 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0830100230 0x7f0830102610 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0818004940 tx=0x7f0818009e30 comp rx=0 tx=0).stop 2024-01-23T08:50:14.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.288+0000 7f08356dd700 1 -- 172.21.15.59:0/847987273 shutdown_connections 2024-01-23T08:50:14.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.288+0000 7f08356dd700 1 --2- 172.21.15.59:0/847987273 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0830100230 0x7f0830102610 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:14.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.288+0000 7f08356dd700 1 --2- 172.21.15.59:0/847987273 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08300fd910 0x7f08300ffcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:14.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.288+0000 7f08356dd700 1 -- 172.21.15.59:0/847987273 >> 172.21.15.59:0/847987273 conn(0x7f08300f7370 msgr2=0x7f08300f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:14.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.288+0000 7f08356dd700 1 -- 172.21.15.59:0/847987273 shutdown_connections 2024-01-23T08:50:14.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.288+0000 7f08356dd700 1 -- 172.21.15.59:0/847987273 wait complete. 2024-01-23T08:50:14.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.289+0000 7f08356dd700 1 Processor -- start 2024-01-23T08:50:14.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.289+0000 7f08356dd700 1 -- start start 2024-01-23T08:50:14.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.289+0000 7f08356dd700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08300fd910 0x7f083010d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:14.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.289+0000 7f08356dd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0830100230 0x7f083010dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:14.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.289+0000 7f08356dd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083010e300 con 0x7f0830100230 2024-01-23T08:50:14.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.289+0000 7f08356dd700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083010eba0 con 0x7f08300fd910 2024-01-23T08:50:14.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.289+0000 7f082e7fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0830100230 0x7f083010dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:14.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.289+0000 7f082e7fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0830100230 0x7f083010dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55358/0 (socket says 172.21.15.59:55358) 2024-01-23T08:50:14.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.289+0000 7f082e7fc700 1 -- 172.21.15.59:0/2932682341 learned_addr learned my addr 172.21.15.59:0/2932682341 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:50:14.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.289+0000 7f082effd700 1 --2- 172.21.15.59:0/2932682341 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08300fd910 0x7f083010d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:14.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.289+0000 7f082e7fc700 1 -- 172.21.15.59:0/2932682341 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08300fd910 msgr2=0x7f083010d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:14.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.289+0000 7f082e7fc700 1 --2- 172.21.15.59:0/2932682341 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08300fd910 0x7f083010d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:14.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.289+0000 7f082e7fc700 1 -- 172.21.15.59:0/2932682341 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0818005040 con 0x7f0830100230 2024-01-23T08:50:14.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.290+0000 7f082e7fc700 1 --2- 172.21.15.59:0/2932682341 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0830100230 0x7f083010dd00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0818009360 tx=0x7f0818009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:14.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.290+0000 7f08277fe700 1 -- 172.21.15.59:0/2932682341 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f081800e040 con 0x7f0830100230 2024-01-23T08:50:14.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.290+0000 7f08277fe700 1 -- 172.21.15.59:0/2932682341 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f081800bc50 con 0x7f0830100230 2024-01-23T08:50:14.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.290+0000 7f08277fe700 1 -- 172.21.15.59:0/2932682341 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f081801c3f0 con 0x7f0830100230 2024-01-23T08:50:14.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.290+0000 7f08356dd700 1 -- 172.21.15.59:0/2932682341 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f083010ee20 con 0x7f0830100230 2024-01-23T08:50:14.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.290+0000 7f08356dd700 1 -- 172.21.15.59:0/2932682341 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f083010f360 con 0x7f0830100230 2024-01-23T08:50:14.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.291+0000 7f08356dd700 1 -- 172.21.15.59:0/2932682341 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08301076d0 con 0x7f0830100230 2024-01-23T08:50:14.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.291+0000 7f08277fe700 1 -- 172.21.15.59:0/2932682341 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f0818013420 con 0x7f0830100230 2024-01-23T08:50:14.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.292+0000 7f08277fe700 1 --2- 172.21.15.59:0/2932682341 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f081c068f60 0x7f081c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:14.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.292+0000 7f08277fe700 1 -- 172.21.15.59:0/2932682341 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0818088200 con 0x7f0830100230 2024-01-23T08:50:14.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.292+0000 7f082effd700 1 --2- 172.21.15.59:0/2932682341 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f081c068f60 0x7f081c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:14.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.292+0000 7f082effd700 1 --2- 172.21.15.59:0/2932682341 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f081c068f60 0x7f081c06b410 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0820007a40 tx=0x7f0820009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:14.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.296+0000 7f08277fe700 1 -- 172.21.15.59:0/2932682341 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0818057ff0 con 0x7f0830100230 2024-01-23T08:50:14.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:14 smithi059 ceph-mon[28424]: mds.? [v2:172.21.15.59:6828/4118577293,v1:172.21.15.59:6829/4118577293] up:resolve 2024-01-23T08:50:14.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:14 smithi059 ceph-mon[28424]: mds.? [v2:172.21.15.59:6826/3959991874,v1:172.21.15.59:6827/3959991874] up:boot 2024-01-23T08:50:14.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:14 smithi059 ceph-mon[28424]: fsmap cephfs:2/2 {0=cephfs.smithi059.twpxfz=up:resolve,1=cephfs.smithi137.mgacmp=up:active} 1 up:standby-replay 1 up:standby 2024-01-23T08:50:14.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:14 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi059.lsxsmm"}]: dispatch 2024-01-23T08:50:14.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.540+0000 7f08356dd700 1 -- 172.21.15.59:0/2932682341 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0830105830 con 0x7f0830100230 2024-01-23T08:50:14.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.541+0000 7f08277fe700 1 -- 172.21.15.59:0/2932682341 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1834 (secure 0 0 0) 0x7f0818057b80 con 0x7f0830100230 2024-01-23T08:50:14.538 INFO:teuthology.orchestra.run.smithi059.stdout:e16 2024-01-23T08:50:14.538 INFO:teuthology.orchestra.run.smithi059.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-01-23T08:50:14.538 INFO:teuthology.orchestra.run.smithi059.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-23T08:50:14.538 INFO:teuthology.orchestra.run.smithi059.stdout:legacy client fscid: 1 2024-01-23T08:50:14.538 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:50:14.538 INFO:teuthology.orchestra.run.smithi059.stdout:Filesystem 'cephfs' (1) 2024-01-23T08:50:14.538 INFO:teuthology.orchestra.run.smithi059.stdout:fs_name cephfs 2024-01-23T08:50:14.539 INFO:teuthology.orchestra.run.smithi059.stdout:epoch 16 2024-01-23T08:50:14.539 INFO:teuthology.orchestra.run.smithi059.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-01-23T08:50:14.539 INFO:teuthology.orchestra.run.smithi059.stdout:created 2024-01-23T08:49:56.450575+0000 2024-01-23T08:50:14.539 INFO:teuthology.orchestra.run.smithi059.stdout:modified 2024-01-23T08:50:14.071934+0000 2024-01-23T08:50:14.539 INFO:teuthology.orchestra.run.smithi059.stdout:tableserver 0 2024-01-23T08:50:14.539 INFO:teuthology.orchestra.run.smithi059.stdout:root 0 2024-01-23T08:50:14.539 INFO:teuthology.orchestra.run.smithi059.stdout:session_timeout 60 2024-01-23T08:50:14.539 INFO:teuthology.orchestra.run.smithi059.stdout:session_autoclose 300 2024-01-23T08:50:14.539 INFO:teuthology.orchestra.run.smithi059.stdout:max_file_size 1099511627776 2024-01-23T08:50:14.539 INFO:teuthology.orchestra.run.smithi059.stdout:required_client_features {} 2024-01-23T08:50:14.539 INFO:teuthology.orchestra.run.smithi059.stdout:last_failure 0 2024-01-23T08:50:14.539 INFO:teuthology.orchestra.run.smithi059.stdout:last_failure_osd_epoch 47 2024-01-23T08:50:14.539 INFO:teuthology.orchestra.run.smithi059.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-23T08:50:14.539 INFO:teuthology.orchestra.run.smithi059.stdout:max_mds 2 2024-01-23T08:50:14.539 INFO:teuthology.orchestra.run.smithi059.stdout:in 0,1 2024-01-23T08:50:14.539 INFO:teuthology.orchestra.run.smithi059.stdout:up {0=14480,1=24269} 2024-01-23T08:50:14.540 INFO:teuthology.orchestra.run.smithi059.stdout:failed 2024-01-23T08:50:14.540 INFO:teuthology.orchestra.run.smithi059.stdout:damaged 2024-01-23T08:50:14.540 INFO:teuthology.orchestra.run.smithi059.stdout:stopped 2024-01-23T08:50:14.540 INFO:teuthology.orchestra.run.smithi059.stdout:data_pools [3] 2024-01-23T08:50:14.540 INFO:teuthology.orchestra.run.smithi059.stdout:metadata_pool 2 2024-01-23T08:50:14.540 INFO:teuthology.orchestra.run.smithi059.stdout:inline_data enabled 2024-01-23T08:50:14.540 INFO:teuthology.orchestra.run.smithi059.stdout:balancer 2024-01-23T08:50:14.540 INFO:teuthology.orchestra.run.smithi059.stdout:standby_count_wanted 1 2024-01-23T08:50:14.540 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi059.twpxfz{0:14480} state up:reconnect seq 4 join_fscid=1 addr [v2:172.21.15.59:6828/4118577293,v1:172.21.15.59:6829/4118577293] compat {c=[1],r=[1],i=[7ff]}] 2024-01-23T08:50:14.540 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi137.mgacmp{1:24269} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.137:6824/1814038594,v1:172.21.15.137:6825/1814038594] compat {c=[1],r=[1],i=[7ff]}] 2024-01-23T08:50:14.540 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi137.scidwx{1:24277} state up:standby-replay seq 1 addr [v2:172.21.15.137:6826/839018117,v1:172.21.15.137:6827/839018117] compat {c=[1],r=[1],i=[7ff]}] 2024-01-23T08:50:14.540 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:50:14.540 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:50:14.540 INFO:teuthology.orchestra.run.smithi059.stdout:Standby daemons: 2024-01-23T08:50:14.541 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:50:14.541 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi059.lsxsmm{-1:14492} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.59:6826/3959991874,v1:172.21.15.59:6827/3959991874] compat {c=[1],r=[1],i=[7ff]}] 2024-01-23T08:50:14.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.543+0000 7f08356dd700 1 -- 172.21.15.59:0/2932682341 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f081c068f60 msgr2=0x7f081c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:14.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.543+0000 7f08356dd700 1 --2- 172.21.15.59:0/2932682341 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f081c068f60 0x7f081c06b410 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0820007a40 tx=0x7f0820009da0 comp rx=0 tx=0).stop 2024-01-23T08:50:14.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.543+0000 7f08356dd700 1 -- 172.21.15.59:0/2932682341 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0830100230 msgr2=0x7f083010dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:14.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.543+0000 7f08356dd700 1 --2- 172.21.15.59:0/2932682341 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0830100230 0x7f083010dd00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0818009360 tx=0x7f0818009e30 comp rx=0 tx=0).stop 2024-01-23T08:50:14.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.543+0000 7f08356dd700 1 -- 172.21.15.59:0/2932682341 shutdown_connections 2024-01-23T08:50:14.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.543+0000 7f08356dd700 1 --2- 172.21.15.59:0/2932682341 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f081c068f60 0x7f081c06b410 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:14.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.543+0000 7f08356dd700 1 --2- 172.21.15.59:0/2932682341 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08300fd910 0x7f083010d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:14.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.543+0000 7f08356dd700 1 --2- 172.21.15.59:0/2932682341 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0830100230 0x7f083010dd00 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:14.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.543+0000 7f08356dd700 1 -- 172.21.15.59:0/2932682341 >> 172.21.15.59:0/2932682341 conn(0x7f08300f7370 msgr2=0x7f08300f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:14.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.543+0000 7f08356dd700 1 -- 172.21.15.59:0/2932682341 shutdown_connections 2024-01-23T08:50:14.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:14.543+0000 7f08356dd700 1 -- 172.21.15.59:0/2932682341 wait complete. 2024-01-23T08:50:14.542 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 16 2024-01-23T08:50:14.542 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:14 smithi137 ceph-mon[35787]: mds.? [v2:172.21.15.59:6828/4118577293,v1:172.21.15.59:6829/4118577293] up:resolve 2024-01-23T08:50:14.542 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:14 smithi137 ceph-mon[35787]: mds.? [v2:172.21.15.59:6826/3959991874,v1:172.21.15.59:6827/3959991874] up:boot 2024-01-23T08:50:14.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:14 smithi137 ceph-mon[35787]: fsmap cephfs:2/2 {0=cephfs.smithi059.twpxfz=up:resolve,1=cephfs.smithi137.mgacmp=up:active} 1 up:standby-replay 1 up:standby 2024-01-23T08:50:14.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:14 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi059.lsxsmm"}]: dispatch 2024-01-23T08:50:15.195 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:15 smithi059 ceph-mon[28424]: mds.? [v2:172.21.15.59:6828/4118577293,v1:172.21.15.59:6829/4118577293] up:reconnect 2024-01-23T08:50:15.195 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:15 smithi059 ceph-mon[28424]: fsmap cephfs:2/2 {0=cephfs.smithi059.twpxfz=up:reconnect,1=cephfs.smithi137.mgacmp=up:active} 1 up:standby-replay 1 up:standby 2024-01-23T08:50:15.196 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:15 smithi059 ceph-mon[28424]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-01-23T08:50:15.196 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:15 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/2932682341' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-01-23T08:50:15.220 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-01-23T08:50:15.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:15 smithi137 ceph-mon[35787]: mds.? [v2:172.21.15.59:6828/4118577293,v1:172.21.15.59:6829/4118577293] up:reconnect 2024-01-23T08:50:15.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:15 smithi137 ceph-mon[35787]: fsmap cephfs:2/2 {0=cephfs.smithi059.twpxfz=up:reconnect,1=cephfs.smithi137.mgacmp=up:active} 1 up:standby-replay 1 up:standby 2024-01-23T08:50:15.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:15 smithi137 ceph-mon[35787]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-01-23T08:50:15.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:15 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/2932682341' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-01-23T08:50:15.676 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:50:16.159 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:16 smithi059 ceph-mon[28424]: mds.? [v2:172.21.15.59:6828/4118577293,v1:172.21.15.59:6829/4118577293] up:rejoin 2024-01-23T08:50:16.159 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:16 smithi059 ceph-mon[28424]: mds.? [v2:172.21.15.137:6826/839018117,v1:172.21.15.137:6827/839018117] up:standby-replay 2024-01-23T08:50:16.159 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:16 smithi059 ceph-mon[28424]: fsmap cephfs:2/2 {0=cephfs.smithi059.twpxfz=up:rejoin,1=cephfs.smithi137.mgacmp=up:active} 1 up:standby-replay 1 up:standby 2024-01-23T08:50:16.159 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:16 smithi059 ceph-mon[28424]: daemon mds.cephfs.smithi059.twpxfz is now active in filesystem cephfs as rank 0 2024-01-23T08:50:16.159 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:16 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:16.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:16 smithi137 ceph-mon[35787]: mds.? [v2:172.21.15.59:6828/4118577293,v1:172.21.15.59:6829/4118577293] up:rejoin 2024-01-23T08:50:16.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:16 smithi137 ceph-mon[35787]: mds.? [v2:172.21.15.137:6826/839018117,v1:172.21.15.137:6827/839018117] up:standby-replay 2024-01-23T08:50:16.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:16 smithi137 ceph-mon[35787]: fsmap cephfs:2/2 {0=cephfs.smithi059.twpxfz=up:rejoin,1=cephfs.smithi137.mgacmp=up:active} 1 up:standby-replay 1 up:standby 2024-01-23T08:50:16.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:16 smithi137 ceph-mon[35787]: daemon mds.cephfs.smithi059.twpxfz is now active in filesystem cephfs as rank 0 2024-01-23T08:50:16.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:16 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:17.274 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:17 smithi059 ceph-mon[28424]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-01-23T08:50:17.274 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:17 smithi059 ceph-mon[28424]: mds.? [v2:172.21.15.59:6828/4118577293,v1:172.21.15.59:6829/4118577293] up:active 2024-01-23T08:50:17.275 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:17 smithi059 ceph-mon[28424]: fsmap cephfs:2 {0=cephfs.smithi059.twpxfz=up:active,1=cephfs.smithi137.mgacmp=up:active} 1 up:standby-replay 1 up:standby 2024-01-23T08:50:17.275 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:17 smithi059 ceph-mon[28424]: fsmap cephfs:2 {0=cephfs.smithi059.twpxfz=up:active,1=cephfs.smithi137.mgacmp=up:active} 2 up:standby-replay 2024-01-23T08:50:17.275 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:17 smithi059 ceph-mon[28424]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.9 KiB/s rd, 614 B/s wr, 5 op/s 2024-01-23T08:50:17.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:17 smithi137 ceph-mon[35787]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-01-23T08:50:17.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:17 smithi137 ceph-mon[35787]: mds.? [v2:172.21.15.59:6828/4118577293,v1:172.21.15.59:6829/4118577293] up:active 2024-01-23T08:50:17.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:17 smithi137 ceph-mon[35787]: fsmap cephfs:2 {0=cephfs.smithi059.twpxfz=up:active,1=cephfs.smithi137.mgacmp=up:active} 1 up:standby-replay 1 up:standby 2024-01-23T08:50:17.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:17 smithi137 ceph-mon[35787]: fsmap cephfs:2 {0=cephfs.smithi059.twpxfz=up:active,1=cephfs.smithi137.mgacmp=up:active} 2 up:standby-replay 2024-01-23T08:50:17.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:17 smithi137 ceph-mon[35787]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.9 KiB/s rd, 614 B/s wr, 5 op/s 2024-01-23T08:50:18.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.459+0000 7f2a9ebac700 1 -- 172.21.15.59:0/3706967096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a980fd910 msgr2=0x7f2a980ffcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:18.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.459+0000 7f2a9ebac700 1 --2- 172.21.15.59:0/3706967096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a980fd910 0x7f2a980ffcf0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c004940 tx=0x7f2a8c009e30 comp rx=0 tx=0).stop 2024-01-23T08:50:18.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.459+0000 7f2a9ebac700 1 -- 172.21.15.59:0/3706967096 shutdown_connections 2024-01-23T08:50:18.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.459+0000 7f2a9ebac700 1 --2- 172.21.15.59:0/3706967096 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2a98100230 0x7f2a98102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:18.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.459+0000 7f2a9ebac700 1 --2- 172.21.15.59:0/3706967096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a980fd910 0x7f2a980ffcf0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:18.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.459+0000 7f2a9ebac700 1 -- 172.21.15.59:0/3706967096 >> 172.21.15.59:0/3706967096 conn(0x7f2a980f7370 msgr2=0x7f2a980f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:18.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.459+0000 7f2a9ebac700 1 -- 172.21.15.59:0/3706967096 shutdown_connections 2024-01-23T08:50:18.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.459+0000 7f2a9ebac700 1 -- 172.21.15.59:0/3706967096 wait complete. 2024-01-23T08:50:18.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.459+0000 7f2a9ebac700 1 Processor -- start 2024-01-23T08:50:18.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.459+0000 7f2a9ebac700 1 -- start start 2024-01-23T08:50:18.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.460+0000 7f2a9ebac700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2a980fd910 0x7f2a980fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:18.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.460+0000 7f2a9ebac700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a98100230 0x7f2a980fb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:18.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.460+0000 7f2a9ebac700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a980fbeb0 con 0x7f2a98100230 2024-01-23T08:50:18.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.460+0000 7f2a9ebac700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a980fc020 con 0x7f2a980fd910 2024-01-23T08:50:18.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.460+0000 7f2a97fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a98100230 0x7f2a980fb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:18.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.460+0000 7f2a97fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a98100230 0x7f2a980fb820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48292/0 (socket says 172.21.15.59:48292) 2024-01-23T08:50:18.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.460+0000 7f2a97fff700 1 -- 172.21.15.59:0/4020524903 learned_addr learned my addr 172.21.15.59:0/4020524903 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:50:18.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.460+0000 7f2a9c948700 1 --2- 172.21.15.59:0/4020524903 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2a980fd910 0x7f2a980fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:18.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.460+0000 7f2a97fff700 1 -- 172.21.15.59:0/4020524903 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2a980fd910 msgr2=0x7f2a980fd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:18.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.460+0000 7f2a97fff700 1 --2- 172.21.15.59:0/4020524903 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2a980fd910 0x7f2a980fd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:18.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.460+0000 7f2a97fff700 1 -- 172.21.15.59:0/4020524903 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8c005040 con 0x7f2a98100230 2024-01-23T08:50:18.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.460+0000 7f2a97fff700 1 --2- 172.21.15.59:0/4020524903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a98100230 0x7f2a980fb820 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2a8400b790 tx=0x7f2a84009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:18.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.461+0000 7f2a957fa700 1 -- 172.21.15.59:0/4020524903 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a8400f770 con 0x7f2a98100230 2024-01-23T08:50:18.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.461+0000 7f2a957fa700 1 -- 172.21.15.59:0/4020524903 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a8400fd80 con 0x7f2a98100230 2024-01-23T08:50:18.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.461+0000 7f2a9ebac700 1 -- 172.21.15.59:0/4020524903 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a980fc300 con 0x7f2a98100230 2024-01-23T08:50:18.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.461+0000 7f2a957fa700 1 -- 172.21.15.59:0/4020524903 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a8400d310 con 0x7f2a98100230 2024-01-23T08:50:18.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.461+0000 7f2a9ebac700 1 -- 172.21.15.59:0/4020524903 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a980fc680 con 0x7f2a98100230 2024-01-23T08:50:18.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.462+0000 7f2a9ebac700 1 -- 172.21.15.59:0/4020524903 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a9804bd60 con 0x7f2a98100230 2024-01-23T08:50:18.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.462+0000 7f2a957fa700 1 -- 172.21.15.59:0/4020524903 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f2a84010420 con 0x7f2a98100230 2024-01-23T08:50:18.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.462+0000 7f2a957fa700 1 --2- 172.21.15.59:0/4020524903 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f2a88068f70 0x7f2a8806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:18.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.463+0000 7f2a957fa700 1 -- 172.21.15.59:0/4020524903 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2a84086cb0 con 0x7f2a98100230 2024-01-23T08:50:18.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.463+0000 7f2a9c948700 1 --2- 172.21.15.59:0/4020524903 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f2a88068f70 0x7f2a8806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:18.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.463+0000 7f2a9c948700 1 --2- 172.21.15.59:0/4020524903 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f2a88068f70 0x7f2a8806b420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00b3b0 tx=0x7f2a8c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:18.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.467+0000 7f2a957fa700 1 -- 172.21.15.59:0/4020524903 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2a84056c60 con 0x7f2a98100230 2024-01-23T08:50:18.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.720+0000 7f2a9ebac700 1 -- 172.21.15.59:0/4020524903 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a98115be0 con 0x7f2a98100230 2024-01-23T08:50:18.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.721+0000 7f2a957fa700 1 -- 172.21.15.59:0/4020524903 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4722 (secure 0 0 0) 0x7f2a840567f0 con 0x7f2a98100230 2024-01-23T08:50:18.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.723+0000 7f2a9ebac700 1 -- 172.21.15.59:0/4020524903 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f2a88068f70 msgr2=0x7f2a8806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:18.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.723+0000 7f2a9ebac700 1 --2- 172.21.15.59:0/4020524903 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f2a88068f70 0x7f2a8806b420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00b3b0 tx=0x7f2a8c009e30 comp rx=0 tx=0).stop 2024-01-23T08:50:18.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.723+0000 7f2a9ebac700 1 -- 172.21.15.59:0/4020524903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a98100230 msgr2=0x7f2a980fb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:18.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.723+0000 7f2a9ebac700 1 --2- 172.21.15.59:0/4020524903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a98100230 0x7f2a980fb820 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2a8400b790 tx=0x7f2a84009250 comp rx=0 tx=0).stop 2024-01-23T08:50:18.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.723+0000 7f2a9ebac700 1 -- 172.21.15.59:0/4020524903 shutdown_connections 2024-01-23T08:50:18.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.723+0000 7f2a9ebac700 1 --2- 172.21.15.59:0/4020524903 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f2a88068f70 0x7f2a8806b420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:18.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.723+0000 7f2a9ebac700 1 --2- 172.21.15.59:0/4020524903 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2a980fd910 0x7f2a980fd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:18.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.723+0000 7f2a9ebac700 1 --2- 172.21.15.59:0/4020524903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a98100230 0x7f2a980fb820 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:18.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.723+0000 7f2a9ebac700 1 -- 172.21.15.59:0/4020524903 >> 172.21.15.59:0/4020524903 conn(0x7f2a980f7370 msgr2=0x7f2a980f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:18.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.723+0000 7f2a9ebac700 1 -- 172.21.15.59:0/4020524903 shutdown_connections 2024-01-23T08:50:18.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:18.723+0000 7f2a9ebac700 1 -- 172.21.15.59:0/4020524903 wait complete. 2024-01-23T08:50:18.721 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 19 2024-01-23T08:50:18.730 INFO:teuthology.orchestra.run.smithi059.stdout:true 2024-01-23T08:50:19.358 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-01-23T08:50:19.690 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:19 smithi059 ceph-mon[28424]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 6 op/s 2024-01-23T08:50:19.690 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:19 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/4020524903' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-23T08:50:19.691 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:19 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:19.691 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:19 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:19.691 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:19 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:50:19.691 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:19 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:50:19.691 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:19 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:19.691 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:19 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:50:19.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:19 smithi137 ceph-mon[35787]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 6 op/s 2024-01-23T08:50:19.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:19 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/4020524903' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-23T08:50:19.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:19 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:19.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:19 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:19.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:19 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:50:19.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:19 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:50:19.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:19 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:19.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:19 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:50:20.177 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:50:21.312 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:21 smithi059 ceph-mon[28424]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 6 op/s 2024-01-23T08:50:21.312 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:21 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:21.312 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:21 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:21.542 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:21 smithi137 ceph-mon[35787]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 6 op/s 2024-01-23T08:50:21.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:21 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:21.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:21 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:22.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.182+0000 7faf12581700 1 -- 172.21.15.59:0/1535347587 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf0c0fdfe0 msgr2=0x7faf0c0fe430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:22.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.182+0000 7faf12581700 1 --2- 172.21.15.59:0/1535347587 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf0c0fdfe0 0x7faf0c0fe430 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7faefc004940 tx=0x7faefc009e30 comp rx=0 tx=0).stop 2024-01-23T08:50:22.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.182+0000 7faf12581700 1 -- 172.21.15.59:0/1535347587 shutdown_connections 2024-01-23T08:50:22.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.182+0000 7faf12581700 1 --2- 172.21.15.59:0/1535347587 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf0c0fdfe0 0x7faf0c0fe430 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:22.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.182+0000 7faf12581700 1 --2- 172.21.15.59:0/1535347587 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf0c0fcdd0 0x7faf0c0fd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:22.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.182+0000 7faf12581700 1 -- 172.21.15.59:0/1535347587 >> 172.21.15.59:0/1535347587 conn(0x7faf0c0f84b0 msgr2=0x7faf0c0fa8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:22.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.182+0000 7faf12581700 1 -- 172.21.15.59:0/1535347587 shutdown_connections 2024-01-23T08:50:22.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.182+0000 7faf12581700 1 -- 172.21.15.59:0/1535347587 wait complete. 2024-01-23T08:50:22.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.183+0000 7faf12581700 1 Processor -- start 2024-01-23T08:50:22.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.183+0000 7faf12581700 1 -- start start 2024-01-23T08:50:22.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.183+0000 7faf12581700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf0c0fcdd0 0x7faf0c071cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:22.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.183+0000 7faf12581700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf0c0fdfe0 0x7faf0c070170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:22.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.183+0000 7faf12581700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0c070770 con 0x7faf0c0fcdd0 2024-01-23T08:50:22.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.183+0000 7faf12581700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0c0708e0 con 0x7faf0c0fdfe0 2024-01-23T08:50:22.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.183+0000 7faf0bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf0c0fcdd0 0x7faf0c071cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:22.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.183+0000 7faf0bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf0c0fcdd0 0x7faf0c071cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48316/0 (socket says 172.21.15.59:48316) 2024-01-23T08:50:22.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.183+0000 7faf0bfff700 1 -- 172.21.15.59:0/1144588447 learned_addr learned my addr 172.21.15.59:0/1144588447 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:50:22.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.183+0000 7faf0b7fe700 1 --2- 172.21.15.59:0/1144588447 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf0c0fdfe0 0x7faf0c070170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:22.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.184+0000 7faf0bfff700 1 -- 172.21.15.59:0/1144588447 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf0c0fdfe0 msgr2=0x7faf0c070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:22.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.184+0000 7faf0bfff700 1 --2- 172.21.15.59:0/1144588447 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf0c0fdfe0 0x7faf0c070170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:22.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.184+0000 7faf0bfff700 1 -- 172.21.15.59:0/1144588447 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faefc005040 con 0x7faf0c0fcdd0 2024-01-23T08:50:22.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.184+0000 7faf0b7fe700 1 --2- 172.21.15.59:0/1144588447 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf0c0fdfe0 0x7faf0c070170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-23T08:50:22.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.184+0000 7faf0bfff700 1 --2- 172.21.15.59:0/1144588447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf0c0fcdd0 0x7faf0c071cf0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7faef4002a80 tx=0x7faef4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:22.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.184+0000 7faf08ff9700 1 -- 172.21.15.59:0/1144588447 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faef4007cc0 con 0x7faf0c0fcdd0 2024-01-23T08:50:22.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.184+0000 7faf12581700 1 -- 172.21.15.59:0/1144588447 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf0c070bc0 con 0x7faf0c0fcdd0 2024-01-23T08:50:22.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.184+0000 7faf08ff9700 1 -- 172.21.15.59:0/1144588447 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faef400f040 con 0x7faf0c0fcdd0 2024-01-23T08:50:22.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.184+0000 7faf08ff9700 1 -- 172.21.15.59:0/1144588447 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faef400da20 con 0x7faf0c0fcdd0 2024-01-23T08:50:22.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.184+0000 7faf12581700 1 -- 172.21.15.59:0/1144588447 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf0c1157d0 con 0x7faf0c0fcdd0 2024-01-23T08:50:22.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.185+0000 7faf12581700 1 -- 172.21.15.59:0/1144588447 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf0c1076e0 con 0x7faf0c0fcdd0 2024-01-23T08:50:22.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.185+0000 7faf08ff9700 1 -- 172.21.15.59:0/1144588447 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7faef400db80 con 0x7faf0c0fcdd0 2024-01-23T08:50:22.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.186+0000 7faf08ff9700 1 --2- 172.21.15.59:0/1144588447 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7faef8068f60 0x7faef806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:22.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.186+0000 7faf08ff9700 1 -- 172.21.15.59:0/1144588447 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7faef4086bf0 con 0x7faf0c0fcdd0 2024-01-23T08:50:22.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.186+0000 7faf0b7fe700 1 --2- 172.21.15.59:0/1144588447 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7faef8068f60 0x7faef806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:22.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.186+0000 7faf0b7fe700 1 --2- 172.21.15.59:0/1144588447 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7faef8068f60 0x7faef806b410 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7faefc004560 tx=0x7faefc009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:22.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.190+0000 7faf08ff9700 1 -- 172.21.15.59:0/1144588447 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7faef4056a80 con 0x7faf0c0fcdd0 2024-01-23T08:50:22.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.468+0000 7faf12581700 1 -- 172.21.15.59:0/1144588447 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7faf0c04bd60 con 0x7faf0c0fcdd0 2024-01-23T08:50:22.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.468+0000 7faf08ff9700 1 -- 172.21.15.59:0/1144588447 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v19) v1 ==== 78+0+99 (secure 0 0 0) 0x7faef4056610 con 0x7faf0c0fcdd0 2024-01-23T08:50:22.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.470+0000 7faf12581700 1 -- 172.21.15.59:0/1144588447 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7faef8068f60 msgr2=0x7faef806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:22.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.470+0000 7faf12581700 1 --2- 172.21.15.59:0/1144588447 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7faef8068f60 0x7faef806b410 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7faefc004560 tx=0x7faefc009e30 comp rx=0 tx=0).stop 2024-01-23T08:50:22.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.470+0000 7faf12581700 1 -- 172.21.15.59:0/1144588447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf0c0fcdd0 msgr2=0x7faf0c071cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:22.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.470+0000 7faf12581700 1 --2- 172.21.15.59:0/1144588447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf0c0fcdd0 0x7faf0c071cf0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7faef4002a80 tx=0x7faef4004600 comp rx=0 tx=0).stop 2024-01-23T08:50:22.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.470+0000 7faf12581700 1 -- 172.21.15.59:0/1144588447 shutdown_connections 2024-01-23T08:50:22.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.470+0000 7faf12581700 1 --2- 172.21.15.59:0/1144588447 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7faef8068f60 0x7faef806b410 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:22.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.470+0000 7faf12581700 1 --2- 172.21.15.59:0/1144588447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf0c0fcdd0 0x7faf0c071cf0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:22.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.470+0000 7faf12581700 1 --2- 172.21.15.59:0/1144588447 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf0c0fdfe0 0x7faf0c070170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:22.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.470+0000 7faf12581700 1 -- 172.21.15.59:0/1144588447 >> 172.21.15.59:0/1144588447 conn(0x7faf0c0f84b0 msgr2=0x7faf0c0fa850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:22.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.470+0000 7faf12581700 1 -- 172.21.15.59:0/1144588447 shutdown_connections 2024-01-23T08:50:22.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:22.470+0000 7faf12581700 1 -- 172.21.15.59:0/1144588447 wait complete. 2024-01-23T08:50:22.477 INFO:teuthology.orchestra.run.smithi059.stdout:true 2024-01-23T08:50:23.186 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-01-23T08:50:23.201 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-23T08:50:23.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.330+0000 7f8aa40ee640 1 -- 172.21.15.59:0/2501569279 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a9c0fb000 msgr2=0x7f8a9c0fb410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:23.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.330+0000 7f8aa40ee640 1 --2- 172.21.15.59:0/2501569279 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a9c0fb000 0x7f8a9c0fb410 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8a940048d0 tx=0x7f8a94009da0 comp rx=0 tx=0).stop 2024-01-23T08:50:23.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.330+0000 7f8aa40ee640 1 -- 172.21.15.59:0/2501569279 shutdown_connections 2024-01-23T08:50:23.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.330+0000 7f8aa40ee640 1 --2- 172.21.15.59:0/2501569279 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8a9c0fc2d0 0x7f8a9c0fc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:23.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.330+0000 7f8aa40ee640 1 --2- 172.21.15.59:0/2501569279 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a9c0fb000 0x7f8a9c0fb410 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:23.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.330+0000 7f8aa40ee640 1 -- 172.21.15.59:0/2501569279 >> 172.21.15.59:0/2501569279 conn(0x7f8a9c0f67a0 msgr2=0x7f8a9c0f8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:23.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.330+0000 7f8aa40ee640 1 -- 172.21.15.59:0/2501569279 shutdown_connections 2024-01-23T08:50:23.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.330+0000 7f8aa40ee640 1 -- 172.21.15.59:0/2501569279 wait complete. 2024-01-23T08:50:23.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.331+0000 7f8aa40ee640 1 Processor -- start 2024-01-23T08:50:23.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.331+0000 7f8aa40ee640 1 -- start start 2024-01-23T08:50:23.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.331+0000 7f8aa40ee640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a9c0fb000 0x7f8a9c10c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:23.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.331+0000 7f8aa40ee640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8a9c0fc2d0 0x7f8a9c10ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:23.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.331+0000 7f8aa40ee640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c10d490 con 0x7f8a9c0fb000 2024-01-23T08:50:23.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.331+0000 7f8aa40ee640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c111ee0 con 0x7f8a9c0fc2d0 2024-01-23T08:50:23.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.331+0000 7f8aa1e8a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a9c0fb000 0x7f8a9c10c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:23.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.331+0000 7f8aa1e8a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a9c0fb000 0x7f8a9c10c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48336/0 (socket says 172.21.15.59:48336) 2024-01-23T08:50:23.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.331+0000 7f8aa1e8a640 1 -- 172.21.15.59:0/1589853922 learned_addr learned my addr 172.21.15.59:0/1589853922 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:50:23.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.331+0000 7f8aa1689640 1 --2- 172.21.15.59:0/1589853922 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8a9c0fc2d0 0x7f8a9c10ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:23.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.331+0000 7f8aa1e8a640 1 -- 172.21.15.59:0/1589853922 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8a9c0fc2d0 msgr2=0x7f8a9c10ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:23.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.331+0000 7f8aa1e8a640 1 --2- 172.21.15.59:0/1589853922 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8a9c0fc2d0 0x7f8a9c10ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:23.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.331+0000 7f8aa1e8a640 1 -- 172.21.15.59:0/1589853922 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a94005040 con 0x7f8a9c0fb000 2024-01-23T08:50:23.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.331+0000 7f8aa1689640 1 --2- 172.21.15.59:0/1589853922 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8a9c0fc2d0 0x7f8a9c10ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-23T08:50:23.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.331+0000 7f8aa1e8a640 1 --2- 172.21.15.59:0/1589853922 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a9c0fb000 0x7f8a9c10c900 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8a9402bda0 tx=0x7f8a94009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:23.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.332+0000 7f8a9a7fc640 1 -- 172.21.15.59:0/1589853922 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a940094c0 con 0x7f8a9c0fb000 2024-01-23T08:50:23.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.332+0000 7f8a9a7fc640 1 -- 172.21.15.59:0/1589853922 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8a94009620 con 0x7f8a9c0fb000 2024-01-23T08:50:23.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.332+0000 7f8aa40ee640 1 -- 172.21.15.59:0/1589853922 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a9c112020 con 0x7f8a9c0fb000 2024-01-23T08:50:23.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.332+0000 7f8a9a7fc640 1 -- 172.21.15.59:0/1589853922 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a9403b400 con 0x7f8a9c0fb000 2024-01-23T08:50:23.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.332+0000 7f8aa40ee640 1 -- 172.21.15.59:0/1589853922 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a9c112590 con 0x7f8a9c0fb000 2024-01-23T08:50:23.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.333+0000 7f8aa40ee640 1 -- 172.21.15.59:0/1589853922 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a9c103a30 con 0x7f8a9c0fb000 2024-01-23T08:50:23.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.333+0000 7f8a9a7fc640 1 -- 172.21.15.59:0/1589853922 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f8a9402f070 con 0x7f8a9c0fb000 2024-01-23T08:50:23.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.333+0000 7f8a9a7fc640 1 --2- 172.21.15.59:0/1589853922 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f8a84068ed0 0x7f8a8406b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:23.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.333+0000 7f8a9a7fc640 1 -- 172.21.15.59:0/1589853922 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f8a940a9480 con 0x7f8a9c0fb000 2024-01-23T08:50:23.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.333+0000 7f8aa1689640 1 --2- 172.21.15.59:0/1589853922 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f8a84068ed0 0x7f8a8406b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:23.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.334+0000 7f8aa1689640 1 --2- 172.21.15.59:0/1589853922 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f8a84068ed0 0x7f8a8406b380 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c10de50 tx=0x7f8a9000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:23.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.338+0000 7f8a9a7fc640 1 -- 172.21.15.59:0/1589853922 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f8a94075880 con 0x7f8a9c0fb000 2024-01-23T08:50:23.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.513+0000 7f8aa40ee640 1 -- 172.21.15.59:0/1589853922 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a9c112720 con 0x7f8a9c0fb000 2024-01-23T08:50:23.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.513+0000 7f8a9a7fc640 1 -- 172.21.15.59:0/1589853922 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+14384 (secure 0 0 0) 0x7f8a94078ea0 con 0x7f8a9c0fb000 2024-01-23T08:50:23.511 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:50:23.511 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":47,"fsid":"3fc07a38-b9cb-11ee-95b1-87774f69a715","created":"2024-01-23T08:43:05.316045+0000","modified":"2024-01-23T08:50:12.065364+0000","last_up_change":"2024-01-23T08:48:36.414344+0000","last_in_change":"2024-01-23T08:48:17.747223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-23T08:47:24.641141+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-01-23T08:49:54.816567+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-01-23T08:49:55.680485+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"444a9c2f-e580-443e-9dfe-36843f8c2e13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":2564662034},{"type":"v1","addr":"172.21.15.59:6803","nonce":2564662034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":2564662034},{"type":"v1","addr":"172.21.15.59:6805","nonce":2564662034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":2564662034},{"type":"v1","addr":"172.21.15.59:6809","nonce":2564662034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":2564662034},{"type":"v1","addr":"172.21.15.59:6807","nonce":2564662034}]},"public_addr":"172.21.15.59:6803/2564662034","cluster_addr":"172.21.15.59:6805/2564662034","heartbeat_back_addr":"172.21.15.59:6809/2564662034","heartbeat_front_addr":"172.21.15.59:6807/2564662034","state":["exists","up"]},{"osd":1,"uuid":"c28ec1b9-2f5f-48e6-8322-7092bbe8cdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1673825133},{"type":"v1","addr":"172.21.15.59:6811","nonce":1673825133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1673825133},{"type":"v1","addr":"172.21.15.59:6813","nonce":1673825133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1673825133},{"type":"v1","addr":"172.21.15.59:6817","nonce":1673825133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1673825133},{"type":"v1","addr":"172.21.15.59:6815","nonce":1673825133}]},"public_addr":"172.21.15.59:6811/1673825133","cluster_addr":"172.21.15.59:6813/1673825133","heartbeat_back_addr":"172.21.15.59:6817/1673825133","heartbeat_front_addr":"172.21.15.59:6815/1673825133","state":["exists","up"]},{"osd":2,"uuid":"94d9e85d-6f22-4e83-9f4c-b326e9375e30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":2143964017},{"type":"v1","addr":"172.21.15.59:6819","nonce":2143964017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":2143964017},{"type":"v1","addr":"172.21.15.59:6821","nonce":2143964017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":2143964017},{"type":"v1","addr":"172.21.15.59:6825","nonce":2143964017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":2143964017},{"type":"v1","addr":"172.21.15.59:6823","nonce":2143964017}]},"public_addr":"172.21.15.59:6819/2143964017","cluster_addr":"172.21.15.59:6821/2143964017","heartbeat_back_addr":"172.21.15.59:6825/2143964017","heartbeat_front_addr":"172.21.15.59:6823/2143964017","state":["exists","up"]},{"osd":3,"uuid":"19dfe941-4ce2-4490-8a09-bf85493593bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":2062812031},{"type":"v1","addr":"172.21.15.137:6801","nonce":2062812031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":2062812031},{"type":"v1","addr":"172.21.15.137:6803","nonce":2062812031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":2062812031},{"type":"v1","addr":"172.21.15.137:6807","nonce":2062812031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":2062812031},{"type":"v1","addr":"172.21.15.137:6805","nonce":2062812031}]},"public_addr":"172.21.15.137:6801/2062812031","cluster_addr":"172.21.15.137:6803/2062812031","heartbeat_back_addr":"172.21.15.137:6807/2062812031","heartbeat_front_addr":"172.21.15.137:6805/2062812031","state":["exists","up"]},{"osd":4,"uuid":"a1d6209f-095c-42e2-966a-11559d090531","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":708078156},{"type":"v1","addr":"172.21.15.137:6809","nonce":708078156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":708078156},{"type":"v1","addr":"172.21.15.137:6811","nonce":708078156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":708078156},{"type":"v1","addr":"172.21.15.137:6815","nonce":708078156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":708078156},{"type":"v1","addr":"172.21.15.137:6813","nonce":708078156}]},"public_addr":"172.21.15.137:6809/708078156","cluster_addr":"172.21.15.137:6811/708078156","heartbeat_back_addr":"172.21.15.137:6815/708078156","heartbeat_front_addr":"172.21.15.137:6813/708078156","state":["exists","up"]},{"osd":5,"uuid":"fdaab2d1-80f0-4928-817e-8ac45394367e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":765377139},{"type":"v1","addr":"172.21.15.137:6817","nonce":765377139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":765377139},{"type":"v1","addr":"172.21.15.137:6819","nonce":765377139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":765377139},{"type":"v1","addr":"172.21.15.137:6823","nonce":765377139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":765377139},{"type":"v1","addr":"172.21.15.137:6821","nonce":765377139}]},"public_addr":"172.21.15.137:6817/765377139","cluster_addr":"172.21.15.137:6819/765377139","heartbeat_back_addr":"172.21.15.137:6823/765377139","heartbeat_front_addr":"172.21.15.137:6821/765377139","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-23T08:46:25.064905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-23T08:46:52.940650+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-23T08:47:21.360904+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-23T08:47:43.875916+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-23T08:48:09.457117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-23T08:48:34.887400+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6827/3125523205":"2024-01-24T08:50:12.065334+0000","172.21.15.59:0/715244053":"2024-01-24T08:44:03.688872+0000","172.21.15.59:6800/504555577":"2024-01-24T08:43:31.660943+0000","172.21.15.59:0/2405882784":"2024-01-24T08:45:10.300954+0000","172.21.15.59:6801/504555577":"2024-01-24T08:43:31.660943+0000","172.21.15.59:0/1345586333":"2024-01-24T08:43:31.660943+0000","172.21.15.59:6800/1337467262":"2024-01-24T08:44:03.688872+0000","172.21.15.59:6801/1337467262":"2024-01-24T08:44:03.688872+0000","172.21.15.59:0/3153116091":"2024-01-24T08:44:03.688872+0000","172.21.15.59:0/2596349924":"2024-01-24T08:44:03.688872+0000","172.21.15.59:6800/3614124404":"2024-01-24T08:45:10.300954+0000","172.21.15.59:6801/3614124404":"2024-01-24T08:45:10.300954+0000","172.21.15.59:0/1598234521":"2024-01-24T08:45:10.300954+0000","172.21.15.59:0/1368475691":"2024-01-24T08:43:31.660943+0000","172.21.15.59:0/2423485094":"2024-01-24T08:45:10.300954+0000","172.21.15.59:0/252476549":"2024-01-24T08:43:31.660943+0000","172.21.15.59:6826/3125523205":"2024-01-24T08:50:12.065334+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-23T08:50:23.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.515+0000 7f8aa40ee640 1 -- 172.21.15.59:0/1589853922 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f8a84068ed0 msgr2=0x7f8a8406b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:23.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.515+0000 7f8aa40ee640 1 --2- 172.21.15.59:0/1589853922 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f8a84068ed0 0x7f8a8406b380 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c10de50 tx=0x7f8a9000b040 comp rx=0 tx=0).stop 2024-01-23T08:50:23.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.515+0000 7f8aa40ee640 1 -- 172.21.15.59:0/1589853922 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a9c0fb000 msgr2=0x7f8a9c10c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:23.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.515+0000 7f8aa40ee640 1 --2- 172.21.15.59:0/1589853922 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a9c0fb000 0x7f8a9c10c900 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8a9402bda0 tx=0x7f8a94009da0 comp rx=0 tx=0).stop 2024-01-23T08:50:23.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.515+0000 7f8aa40ee640 1 -- 172.21.15.59:0/1589853922 shutdown_connections 2024-01-23T08:50:23.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.515+0000 7f8aa40ee640 1 --2- 172.21.15.59:0/1589853922 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f8a84068ed0 0x7f8a8406b380 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:23.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.515+0000 7f8aa40ee640 1 --2- 172.21.15.59:0/1589853922 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8a9c0fc2d0 0x7f8a9c10ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:23.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.515+0000 7f8aa40ee640 1 --2- 172.21.15.59:0/1589853922 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a9c0fb000 0x7f8a9c10c900 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:23.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.515+0000 7f8aa40ee640 1 -- 172.21.15.59:0/1589853922 >> 172.21.15.59:0/1589853922 conn(0x7f8a9c0f67a0 msgr2=0x7f8a9c0655a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:23.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.515+0000 7f8aa40ee640 1 -- 172.21.15.59:0/1589853922 shutdown_connections 2024-01-23T08:50:23.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.516+0000 7f8aa40ee640 1 -- 172.21.15.59:0/1589853922 wait complete. 2024-01-23T08:50:23.523 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-23T08:47:24.641141+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-01-23T08:49:54.816567+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-01-23T08:49:55.680485+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-23T08:50:23.524 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-23T08:50:23.542 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:23 smithi137 ceph-mon[35787]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 8 op/s 2024-01-23T08:50:23.542 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:23 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/1144588447' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-01-23T08:50:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.637+0000 7f7be579d640 1 -- 172.21.15.59:0/2699542183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7be00fd1b0 msgr2=0x7f7be00fd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.637+0000 7f7be579d640 1 --2- 172.21.15.59:0/2699542183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7be00fd1b0 0x7f7be00fd620 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7bc80048d0 tx=0x7f7bc8009da0 comp rx=0 tx=0).stop 2024-01-23T08:50:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.637+0000 7f7be579d640 1 -- 172.21.15.59:0/2699542183 shutdown_connections 2024-01-23T08:50:23.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.637+0000 7f7be579d640 1 --2- 172.21.15.59:0/2699542183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7be00fd1b0 0x7f7be00fd620 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:23.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.637+0000 7f7be579d640 1 --2- 172.21.15.59:0/2699542183 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7be00fbf80 0x7f7be00fc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:23.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.637+0000 7f7be579d640 1 -- 172.21.15.59:0/2699542183 >> 172.21.15.59:0/2699542183 conn(0x7f7be00f7680 msgr2=0x7f7be00f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:23.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.637+0000 7f7be579d640 1 -- 172.21.15.59:0/2699542183 shutdown_connections 2024-01-23T08:50:23.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.637+0000 7f7be579d640 1 -- 172.21.15.59:0/2699542183 wait complete. 2024-01-23T08:50:23.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.637+0000 7f7be579d640 1 Processor -- start 2024-01-23T08:50:23.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.638+0000 7f7be579d640 1 -- start start 2024-01-23T08:50:23.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.638+0000 7f7be579d640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7be00fbf80 0x7f7be010c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:23.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.638+0000 7f7be579d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7be00fd1b0 0x7f7be010cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:23.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.638+0000 7f7be579d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be010d2d0 con 0x7f7be00fd1b0 2024-01-23T08:50:23.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.638+0000 7f7be579d640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be0111d20 con 0x7f7be00fbf80 2024-01-23T08:50:23.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.638+0000 7f7bdffff640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7be00fbf80 0x7f7be010c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:23.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.638+0000 7f7bdf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7be00fd1b0 0x7f7be010cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:23.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.638+0000 7f7bdf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7be00fd1b0 0x7f7be010cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48348/0 (socket says 172.21.15.59:48348) 2024-01-23T08:50:23.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.638+0000 7f7bdf7fe640 1 -- 172.21.15.59:0/3501941350 learned_addr learned my addr 172.21.15.59:0/3501941350 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:50:23.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.638+0000 7f7bdffff640 1 -- 172.21.15.59:0/3501941350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7be00fd1b0 msgr2=0x7f7be010cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:23.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.638+0000 7f7bdffff640 1 --2- 172.21.15.59:0/3501941350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7be00fd1b0 0x7f7be010cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:23.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.638+0000 7f7bdffff640 1 -- 172.21.15.59:0/3501941350 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc8005040 con 0x7f7be00fbf80 2024-01-23T08:50:23.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.639+0000 7f7bdffff640 1 --2- 172.21.15.59:0/3501941350 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7be00fbf80 0x7f7be010c740 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7be00fd010 tx=0x7f7bd000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:23.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.639+0000 7f7bdcff9640 1 -- 172.21.15.59:0/3501941350 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bd0007c30 con 0x7f7be00fbf80 2024-01-23T08:50:23.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.639+0000 7f7be579d640 1 -- 172.21.15.59:0/3501941350 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7be0111ec0 con 0x7f7be00fbf80 2024-01-23T08:50:23.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.639+0000 7f7be579d640 1 -- 172.21.15.59:0/3501941350 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7be01123d0 con 0x7f7be00fbf80 2024-01-23T08:50:23.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.639+0000 7f7bdcff9640 1 -- 172.21.15.59:0/3501941350 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7bd000f040 con 0x7f7be00fbf80 2024-01-23T08:50:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.639+0000 7f7bdcff9640 1 -- 172.21.15.59:0/3501941350 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bd000ca40 con 0x7f7be00fbf80 2024-01-23T08:50:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.640+0000 7f7bdcff9640 1 -- 172.21.15.59:0/3501941350 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f7bd000cba0 con 0x7f7be00fbf80 2024-01-23T08:50:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.640+0000 7f7be579d640 1 -- 172.21.15.59:0/3501941350 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7be01049b0 con 0x7f7be00fbf80 2024-01-23T08:50:23.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.641+0000 7f7bdcff9640 1 --2- 172.21.15.59:0/3501941350 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f7bcc068ed0 0x7f7bcc06b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:23.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.641+0000 7f7bdcff9640 1 -- 172.21.15.59:0/3501941350 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7bd0086ff0 con 0x7f7be00fbf80 2024-01-23T08:50:23.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.641+0000 7f7bdf7fe640 1 --2- 172.21.15.59:0/3501941350 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f7bcc068ed0 0x7f7bcc06b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:23.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.641+0000 7f7bdf7fe640 1 --2- 172.21.15.59:0/3501941350 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f7bcc068ed0 0x7f7bcc06b380 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8003080 tx=0x7f7bc8009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:23.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.645+0000 7f7bdcff9640 1 -- 172.21.15.59:0/3501941350 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7bd0053370 con 0x7f7be00fbf80 2024-01-23T08:50:23.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:23 smithi059 ceph-mon[28424]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 8 op/s 2024-01-23T08:50:23.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:23 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/1144588447' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-01-23T08:50:23.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.820+0000 7f7be579d640 1 -- 172.21.15.59:0/3501941350 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7be0112670 con 0x7f7be00fbf80 2024-01-23T08:50:23.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.820+0000 7f7bdcff9640 1 -- 172.21.15.59:0/3501941350 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7bd0056990 con 0x7f7be00fbf80 2024-01-23T08:50:23.818 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-01-23T08:50:23.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.822+0000 7f7be579d640 1 -- 172.21.15.59:0/3501941350 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f7bcc068ed0 msgr2=0x7f7bcc06b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:23.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.822+0000 7f7be579d640 1 --2- 172.21.15.59:0/3501941350 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f7bcc068ed0 0x7f7bcc06b380 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8003080 tx=0x7f7bc8009da0 comp rx=0 tx=0).stop 2024-01-23T08:50:23.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.822+0000 7f7be579d640 1 -- 172.21.15.59:0/3501941350 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7be00fbf80 msgr2=0x7f7be010c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:23.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.822+0000 7f7be579d640 1 --2- 172.21.15.59:0/3501941350 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7be00fbf80 0x7f7be010c740 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7be00fd010 tx=0x7f7bd000b040 comp rx=0 tx=0).stop 2024-01-23T08:50:23.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.822+0000 7f7be579d640 1 -- 172.21.15.59:0/3501941350 shutdown_connections 2024-01-23T08:50:23.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.822+0000 7f7be579d640 1 --2- 172.21.15.59:0/3501941350 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f7bcc068ed0 0x7f7bcc06b380 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:23.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.822+0000 7f7be579d640 1 --2- 172.21.15.59:0/3501941350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7be00fd1b0 0x7f7be010cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:23.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.822+0000 7f7be579d640 1 --2- 172.21.15.59:0/3501941350 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7be00fbf80 0x7f7be010c740 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:23.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.822+0000 7f7be579d640 1 -- 172.21.15.59:0/3501941350 >> 172.21.15.59:0/3501941350 conn(0x7f7be00f7680 msgr2=0x7f7be01003e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:23.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.822+0000 7f7be579d640 1 -- 172.21.15.59:0/3501941350 shutdown_connections 2024-01-23T08:50:23.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.822+0000 7f7be579d640 1 -- 172.21.15.59:0/3501941350 wait complete. 2024-01-23T08:50:23.830 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-01-23T08:50:23.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.955+0000 7f19e179d640 1 -- 172.21.15.59:0/3024377941 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19dc0fc060 msgr2=0x7f19dc0fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:23.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.955+0000 7f19e179d640 1 --2- 172.21.15.59:0/3024377941 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19dc0fc060 0x7f19dc0fc4d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f19c40048d0 tx=0x7f19c4009da0 comp rx=0 tx=0).stop 2024-01-23T08:50:23.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.955+0000 7f19e179d640 1 -- 172.21.15.59:0/3024377941 shutdown_connections 2024-01-23T08:50:23.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.955+0000 7f19e179d640 1 --2- 172.21.15.59:0/3024377941 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19dc0fc060 0x7f19dc0fc4d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:23.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.955+0000 7f19e179d640 1 --2- 172.21.15.59:0/3024377941 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19dc0fae40 0x7f19dc0fb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:23.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.955+0000 7f19e179d640 1 -- 172.21.15.59:0/3024377941 >> 172.21.15.59:0/3024377941 conn(0x7f19dc0f65b0 msgr2=0x7f19dc0f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:23.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.955+0000 7f19e179d640 1 -- 172.21.15.59:0/3024377941 shutdown_connections 2024-01-23T08:50:23.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.955+0000 7f19e179d640 1 -- 172.21.15.59:0/3024377941 wait complete. 2024-01-23T08:50:23.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.956+0000 7f19e179d640 1 Processor -- start 2024-01-23T08:50:23.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.956+0000 7f19e179d640 1 -- start start 2024-01-23T08:50:23.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.956+0000 7f19e179d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19dc0fae40 0x7f19dc064dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:23.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.956+0000 7f19e179d640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19dc0fc060 0x7f19dc1115a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:23.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.956+0000 7f19e179d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc111ae0 con 0x7f19dc0fae40 2024-01-23T08:50:23.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.956+0000 7f19e179d640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc111c20 con 0x7f19dc0fc060 2024-01-23T08:50:23.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.956+0000 7f19dbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19dc0fae40 0x7f19dc064dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:23.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.956+0000 7f19dbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19dc0fae40 0x7f19dc064dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48368/0 (socket says 172.21.15.59:48368) 2024-01-23T08:50:23.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.956+0000 7f19dbfff640 1 -- 172.21.15.59:0/4279511794 learned_addr learned my addr 172.21.15.59:0/4279511794 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:50:23.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.956+0000 7f19db7fe640 1 --2- 172.21.15.59:0/4279511794 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19dc0fc060 0x7f19dc1115a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:23.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.957+0000 7f19dbfff640 1 -- 172.21.15.59:0/4279511794 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19dc0fc060 msgr2=0x7f19dc1115a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:23.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.957+0000 7f19dbfff640 1 --2- 172.21.15.59:0/4279511794 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19dc0fc060 0x7f19dc1115a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:23.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.957+0000 7f19dbfff640 1 -- 172.21.15.59:0/4279511794 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c4005040 con 0x7f19dc0fae40 2024-01-23T08:50:23.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.957+0000 7f19dbfff640 1 --2- 172.21.15.59:0/4279511794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19dc0fae40 0x7f19dc064dc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f19cc00eba0 tx=0x7f19cc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:23.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.957+0000 7f19d8ff9640 1 -- 172.21.15.59:0/4279511794 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19cc007c30 con 0x7f19dc0fae40 2024-01-23T08:50:23.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.957+0000 7f19d8ff9640 1 -- 172.21.15.59:0/4279511794 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f19cc00f040 con 0x7f19dc0fae40 2024-01-23T08:50:23.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.957+0000 7f19e179d640 1 -- 172.21.15.59:0/4279511794 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19dc111ed0 con 0x7f19dc0fae40 2024-01-23T08:50:23.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.957+0000 7f19d8ff9640 1 -- 172.21.15.59:0/4279511794 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19cc0095d0 con 0x7f19dc0fae40 2024-01-23T08:50:23.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.957+0000 7f19e179d640 1 -- 172.21.15.59:0/4279511794 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19dc1123e0 con 0x7f19dc0fae40 2024-01-23T08:50:23.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.958+0000 7f19e179d640 1 -- 172.21.15.59:0/4279511794 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19dc1038b0 con 0x7f19dc0fae40 2024-01-23T08:50:23.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.959+0000 7f19d8ff9640 1 -- 172.21.15.59:0/4279511794 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f19cc00c710 con 0x7f19dc0fae40 2024-01-23T08:50:23.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.959+0000 7f19d8ff9640 1 --2- 172.21.15.59:0/4279511794 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f19c8068ed0 0x7f19c806b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:23.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.959+0000 7f19d8ff9640 1 -- 172.21.15.59:0/4279511794 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f19cc087090 con 0x7f19dc0fae40 2024-01-23T08:50:23.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.959+0000 7f19db7fe640 1 --2- 172.21.15.59:0/4279511794 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f19c8068ed0 0x7f19c806b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:23.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.960+0000 7f19db7fe640 1 --2- 172.21.15.59:0/4279511794 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f19c8068ed0 0x7f19c806b380 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f19dc064b70 tx=0x7f19c4009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:23.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:23.964+0000 7f19d8ff9640 1 -- 172.21.15.59:0/4279511794 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f19cc053490 con 0x7f19dc0fae40 2024-01-23T08:50:24.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.144+0000 7f19e179d640 1 -- 172.21.15.59:0/4279511794 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f19dc1129a0 con 0x7f19dc0fae40 2024-01-23T08:50:24.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.145+0000 7f19d8ff9640 1 -- 172.21.15.59:0/4279511794 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7f19cc056ab0 con 0x7f19dc0fae40 2024-01-23T08:50:24.142 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-01-23T08:50:24.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.146+0000 7f19e179d640 1 -- 172.21.15.59:0/4279511794 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f19c8068ed0 msgr2=0x7f19c806b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:24.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.146+0000 7f19e179d640 1 --2- 172.21.15.59:0/4279511794 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f19c8068ed0 0x7f19c806b380 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f19dc064b70 tx=0x7f19c4009da0 comp rx=0 tx=0).stop 2024-01-23T08:50:24.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.146+0000 7f19e179d640 1 -- 172.21.15.59:0/4279511794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19dc0fae40 msgr2=0x7f19dc064dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:24.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.146+0000 7f19e179d640 1 --2- 172.21.15.59:0/4279511794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19dc0fae40 0x7f19dc064dc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f19cc00eba0 tx=0x7f19cc00b040 comp rx=0 tx=0).stop 2024-01-23T08:50:24.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.146+0000 7f19e179d640 1 -- 172.21.15.59:0/4279511794 shutdown_connections 2024-01-23T08:50:24.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.146+0000 7f19e179d640 1 --2- 172.21.15.59:0/4279511794 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f19c8068ed0 0x7f19c806b380 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:24.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.146+0000 7f19e179d640 1 --2- 172.21.15.59:0/4279511794 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19dc0fc060 0x7f19dc1115a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:24.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.146+0000 7f19e179d640 1 --2- 172.21.15.59:0/4279511794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19dc0fae40 0x7f19dc064dc0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:24.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.147+0000 7f19e179d640 1 -- 172.21.15.59:0/4279511794 >> 172.21.15.59:0/4279511794 conn(0x7f19dc0f65b0 msgr2=0x7f19dc0ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:24.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.147+0000 7f19e179d640 1 -- 172.21.15.59:0/4279511794 shutdown_connections 2024-01-23T08:50:24.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.147+0000 7f19e179d640 1 -- 172.21.15.59:0/4279511794 wait complete. 2024-01-23T08:50:24.154 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-01-23T08:50:24.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.259+0000 7fbe76b9d640 1 -- 172.21.15.59:0/1913182350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe700fae30 msgr2=0x7fbe700fb240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:24.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.259+0000 7fbe76b9d640 1 --2- 172.21.15.59:0/1913182350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe700fae30 0x7fbe700fb240 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c0048d0 tx=0x7fbe6c009da0 comp rx=0 tx=0).stop 2024-01-23T08:50:24.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.259+0000 7fbe76b9d640 1 -- 172.21.15.59:0/1913182350 shutdown_connections 2024-01-23T08:50:24.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.259+0000 7fbe76b9d640 1 --2- 172.21.15.59:0/1913182350 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe700fc050 0x7fbe700fc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:24.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.259+0000 7fbe76b9d640 1 --2- 172.21.15.59:0/1913182350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe700fae30 0x7fbe700fb240 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:24.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.259+0000 7fbe76b9d640 1 -- 172.21.15.59:0/1913182350 >> 172.21.15.59:0/1913182350 conn(0x7fbe700f65a0 msgr2=0x7fbe700f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:24.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.259+0000 7fbe76b9d640 1 -- 172.21.15.59:0/1913182350 shutdown_connections 2024-01-23T08:50:24.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.259+0000 7fbe76b9d640 1 -- 172.21.15.59:0/1913182350 wait complete. 2024-01-23T08:50:24.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.259+0000 7fbe76b9d640 1 Processor -- start 2024-01-23T08:50:24.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.259+0000 7fbe76b9d640 1 -- start start 2024-01-23T08:50:24.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.259+0000 7fbe76b9d640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe700fae30 0x7fbe7010c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:24.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.259+0000 7fbe76b9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe700fc050 0x7fbe7010ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:24.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.259+0000 7fbe76b9d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7010d340 con 0x7fbe700fc050 2024-01-23T08:50:24.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.259+0000 7fbe76b9d640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe70111d90 con 0x7fbe700fae30 2024-01-23T08:50:24.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.260+0000 7fbe7539a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe700fc050 0x7fbe7010ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:24.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.260+0000 7fbe7539a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe700fc050 0x7fbe7010ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48394/0 (socket says 172.21.15.59:48394) 2024-01-23T08:50:24.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.260+0000 7fbe7539a640 1 -- 172.21.15.59:0/3651502973 learned_addr learned my addr 172.21.15.59:0/3651502973 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:50:24.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.260+0000 7fbe75b9b640 1 --2- 172.21.15.59:0/3651502973 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe700fae30 0x7fbe7010c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:24.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.260+0000 7fbe7539a640 1 -- 172.21.15.59:0/3651502973 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe700fae30 msgr2=0x7fbe7010c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:24.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.260+0000 7fbe7539a640 1 --2- 172.21.15.59:0/3651502973 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe700fae30 0x7fbe7010c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:24.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.260+0000 7fbe7539a640 1 -- 172.21.15.59:0/3651502973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6c005040 con 0x7fbe700fc050 2024-01-23T08:50:24.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.260+0000 7fbe7539a640 1 --2- 172.21.15.59:0/3651502973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe700fc050 0x7fbe7010ccf0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000eb70 tx=0x7fbe6000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:24.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.260+0000 7fbe667fc640 1 -- 172.21.15.59:0/3651502973 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe60007c30 con 0x7fbe700fc050 2024-01-23T08:50:24.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.260+0000 7fbe667fc640 1 -- 172.21.15.59:0/3651502973 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe6000f040 con 0x7fbe700fc050 2024-01-23T08:50:24.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.260+0000 7fbe76b9d640 1 -- 172.21.15.59:0/3651502973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe70111ed0 con 0x7fbe700fc050 2024-01-23T08:50:24.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.260+0000 7fbe667fc640 1 -- 172.21.15.59:0/3651502973 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe600094b0 con 0x7fbe700fc050 2024-01-23T08:50:24.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.260+0000 7fbe76b9d640 1 -- 172.21.15.59:0/3651502973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe70112440 con 0x7fbe700fc050 2024-01-23T08:50:24.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.261+0000 7fbe76b9d640 1 -- 172.21.15.59:0/3651502973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe701038a0 con 0x7fbe700fc050 2024-01-23T08:50:24.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.262+0000 7fbe667fc640 1 -- 172.21.15.59:0/3651502973 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fbe60009720 con 0x7fbe700fc050 2024-01-23T08:50:24.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.262+0000 7fbe667fc640 1 --2- 172.21.15.59:0/3651502973 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbe5c068ed0 0x7fbe5c06b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:24.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.262+0000 7fbe667fc640 1 -- 172.21.15.59:0/3651502973 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fbe60086b80 con 0x7fbe700fc050 2024-01-23T08:50:24.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.262+0000 7fbe75b9b640 1 --2- 172.21.15.59:0/3651502973 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbe5c068ed0 0x7fbe5c06b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:24.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.262+0000 7fbe75b9b640 1 --2- 172.21.15.59:0/3651502973 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbe5c068ed0 0x7fbe5c06b380 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c000c00 tx=0x7fbe6c009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:24.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.266+0000 7fbe667fc640 1 -- 172.21.15.59:0/3651502973 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fbe60052f80 con 0x7fbe700fc050 2024-01-23T08:50:24.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.453+0000 7fbe76b9d640 1 -- 172.21.15.59:0/3651502973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fbe70112b30 con 0x7fbe700fc050 2024-01-23T08:50:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.453+0000 7fbe667fc640 1 -- 172.21.15.59:0/3651502973 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7fbe600565a0 con 0x7fbe700fc050 2024-01-23T08:50:24.451 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-01-23T08:50:24.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.455+0000 7fbe76b9d640 1 -- 172.21.15.59:0/3651502973 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbe5c068ed0 msgr2=0x7fbe5c06b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:24.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.455+0000 7fbe76b9d640 1 --2- 172.21.15.59:0/3651502973 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbe5c068ed0 0x7fbe5c06b380 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c000c00 tx=0x7fbe6c009da0 comp rx=0 tx=0).stop 2024-01-23T08:50:24.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.455+0000 7fbe76b9d640 1 -- 172.21.15.59:0/3651502973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe700fc050 msgr2=0x7fbe7010ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:24.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.455+0000 7fbe76b9d640 1 --2- 172.21.15.59:0/3651502973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe700fc050 0x7fbe7010ccf0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000eb70 tx=0x7fbe6000b040 comp rx=0 tx=0).stop 2024-01-23T08:50:24.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.455+0000 7fbe76b9d640 1 -- 172.21.15.59:0/3651502973 shutdown_connections 2024-01-23T08:50:24.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.455+0000 7fbe76b9d640 1 --2- 172.21.15.59:0/3651502973 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbe5c068ed0 0x7fbe5c06b380 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:24.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.455+0000 7fbe76b9d640 1 --2- 172.21.15.59:0/3651502973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe700fc050 0x7fbe7010ccf0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:24.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.455+0000 7fbe76b9d640 1 --2- 172.21.15.59:0/3651502973 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe700fae30 0x7fbe7010c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:24.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.455+0000 7fbe76b9d640 1 -- 172.21.15.59:0/3651502973 >> 172.21.15.59:0/3651502973 conn(0x7fbe700f65a0 msgr2=0x7fbe70065340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:24.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.455+0000 7fbe76b9d640 1 -- 172.21.15.59:0/3651502973 shutdown_connections 2024-01-23T08:50:24.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.455+0000 7fbe76b9d640 1 -- 172.21.15.59:0/3651502973 wait complete. 2024-01-23T08:50:24.464 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-23T08:50:24.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.565+0000 7f0ae9d9d640 1 -- 172.21.15.59:0/2587690638 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ae40fa240 msgr2=0x7f0ae40fa650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:24.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.565+0000 7f0ae9d9d640 1 --2- 172.21.15.59:0/2587690638 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ae40fa240 0x7f0ae40fa650 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0048d0 tx=0x7f0acc009da0 comp rx=0 tx=0).stop 2024-01-23T08:50:24.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.565+0000 7f0ae9d9d640 1 -- 172.21.15.59:0/2587690638 shutdown_connections 2024-01-23T08:50:24.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.565+0000 7f0ae9d9d640 1 --2- 172.21.15.59:0/2587690638 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ae40fd2f0 0x7f0ae40fd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:24.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.565+0000 7f0ae9d9d640 1 --2- 172.21.15.59:0/2587690638 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ae40fa240 0x7f0ae40fa650 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:24.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.565+0000 7f0ae9d9d640 1 -- 172.21.15.59:0/2587690638 >> 172.21.15.59:0/2587690638 conn(0x7f0ae40f5d00 msgr2=0x7f0ae40f8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:24.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.565+0000 7f0ae9d9d640 1 -- 172.21.15.59:0/2587690638 shutdown_connections 2024-01-23T08:50:24.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.565+0000 7f0ae9d9d640 1 -- 172.21.15.59:0/2587690638 wait complete. 2024-01-23T08:50:24.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.566+0000 7f0ae9d9d640 1 Processor -- start 2024-01-23T08:50:24.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.566+0000 7f0ae9d9d640 1 -- start start 2024-01-23T08:50:24.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.566+0000 7f0ae9d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ae40fa240 0x7f0ae410c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:24.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.566+0000 7f0ae9d9d640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ae40fd2f0 0x7f0ae410cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:24.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.566+0000 7f0ae9d9d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae410d330 con 0x7f0ae40fa240 2024-01-23T08:50:24.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.566+0000 7f0ae9d9d640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae4111d80 con 0x7f0ae40fd2f0 2024-01-23T08:50:24.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.566+0000 7f0ae8d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ae40fa240 0x7f0ae410c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:24.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.566+0000 7f0ae8d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ae40fa240 0x7f0ae410c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48422/0 (socket says 172.21.15.59:48422) 2024-01-23T08:50:24.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.566+0000 7f0ae8d9b640 1 -- 172.21.15.59:0/1058892148 learned_addr learned my addr 172.21.15.59:0/1058892148 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:50:24.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.566+0000 7f0ae3fff640 1 --2- 172.21.15.59:0/1058892148 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ae40fd2f0 0x7f0ae410cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:24.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.567+0000 7f0ae8d9b640 1 -- 172.21.15.59:0/1058892148 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ae40fd2f0 msgr2=0x7f0ae410cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:24.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.567+0000 7f0ae8d9b640 1 --2- 172.21.15.59:0/1058892148 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ae40fd2f0 0x7f0ae410cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:24.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.567+0000 7f0ae8d9b640 1 -- 172.21.15.59:0/1058892148 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0acc005040 con 0x7f0ae40fa240 2024-01-23T08:50:24.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.567+0000 7f0ae8d9b640 1 --2- 172.21.15.59:0/1058892148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ae40fa240 0x7f0ae410c7a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0acc02bda0 tx=0x7f0acc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:24.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.567+0000 7f0ae17fa640 1 -- 172.21.15.59:0/1058892148 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0acc0094c0 con 0x7f0ae40fa240 2024-01-23T08:50:24.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.567+0000 7f0ae9d9d640 1 -- 172.21.15.59:0/1058892148 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ae4111ec0 con 0x7f0ae40fa240 2024-01-23T08:50:24.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.567+0000 7f0ae17fa640 1 -- 172.21.15.59:0/1058892148 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0acc009620 con 0x7f0ae40fa240 2024-01-23T08:50:24.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.567+0000 7f0ae17fa640 1 -- 172.21.15.59:0/1058892148 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0acc03b400 con 0x7f0ae40fa240 2024-01-23T08:50:24.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.567+0000 7f0ae9d9d640 1 -- 172.21.15.59:0/1058892148 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ae41123d0 con 0x7f0ae40fa240 2024-01-23T08:50:24.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.568+0000 7f0ae9d9d640 1 -- 172.21.15.59:0/1058892148 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ae41049a0 con 0x7f0ae40fa240 2024-01-23T08:50:24.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.568+0000 7f0ae17fa640 1 -- 172.21.15.59:0/1058892148 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f0acc02f070 con 0x7f0ae40fa240 2024-01-23T08:50:24.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.569+0000 7f0ae17fa640 1 --2- 172.21.15.59:0/1058892148 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f0ad0068ed0 0x7f0ad006b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:24.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.569+0000 7f0ae17fa640 1 -- 172.21.15.59:0/1058892148 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0acc0a9360 con 0x7f0ae40fa240 2024-01-23T08:50:24.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.569+0000 7f0ae3fff640 1 --2- 172.21.15.59:0/1058892148 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f0ad0068ed0 0x7f0ad006b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:24.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.569+0000 7f0ae3fff640 1 --2- 172.21.15.59:0/1058892148 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f0ad0068ed0 0x7f0ad006b380 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0ae410dcf0 tx=0x7f0ad400b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:24.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.573+0000 7f0ae17fa640 1 -- 172.21.15.59:0/1058892148 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0acc075760 con 0x7f0ae40fa240 2024-01-23T08:50:24.681 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:24 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/1589853922' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-23T08:50:24.682 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:24 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/3501941350' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-23T08:50:24.682 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:24 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/4279511794' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-01-23T08:50:24.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.788+0000 7f0ae9d9d640 1 -- 172.21.15.59:0/1058892148 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ae4112560 con 0x7f0ae40fa240 2024-01-23T08:50:24.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.788+0000 7f0ae17fa640 1 -- 172.21.15.59:0/1058892148 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4722 (secure 0 0 0) 0x7f0acc078d80 con 0x7f0ae40fa240 2024-01-23T08:50:24.786 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-01-23T08:50:24.786 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-01-23T08:49:56.450575+0000","modified":"2024-01-23T08:50:16.092382+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14480,"mds_1":24269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14480":{"gid":14480,"name":"cephfs.smithi059.twpxfz","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.59:6829/4118577293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":4118577293},{"type":"v1","addr":"172.21.15.59:6829","nonce":4118577293}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14492":{"gid":14492,"name":"cephfs.smithi059.lsxsmm","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.59:6827/3959991874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":3959991874},{"type":"v1","addr":"172.21.15.59:6827","nonce":3959991874}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24269":{"gid":24269,"name":"cephfs.smithi137.mgacmp","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.137:6825/1814038594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1814038594},{"type":"v1","addr":"172.21.15.137:6825","nonce":1814038594}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24277":{"gid":24277,"name":"cephfs.smithi137.scidwx","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.137:6827/839018117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":839018117},{"type":"v1","addr":"172.21.15.137:6827","nonce":839018117}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-01-23T08:50:24.786 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 19 2024-01-23T08:50:24.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.790+0000 7f0ae9d9d640 1 -- 172.21.15.59:0/1058892148 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f0ad0068ed0 msgr2=0x7f0ad006b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:24.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.790+0000 7f0ae9d9d640 1 --2- 172.21.15.59:0/1058892148 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f0ad0068ed0 0x7f0ad006b380 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0ae410dcf0 tx=0x7f0ad400b040 comp rx=0 tx=0).stop 2024-01-23T08:50:24.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.790+0000 7f0ae9d9d640 1 -- 172.21.15.59:0/1058892148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ae40fa240 msgr2=0x7f0ae410c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:24.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.790+0000 7f0ae9d9d640 1 --2- 172.21.15.59:0/1058892148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ae40fa240 0x7f0ae410c7a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0acc02bda0 tx=0x7f0acc009da0 comp rx=0 tx=0).stop 2024-01-23T08:50:24.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.790+0000 7f0ae9d9d640 1 -- 172.21.15.59:0/1058892148 shutdown_connections 2024-01-23T08:50:24.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.790+0000 7f0ae9d9d640 1 --2- 172.21.15.59:0/1058892148 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f0ad0068ed0 0x7f0ad006b380 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:24.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.790+0000 7f0ae9d9d640 1 --2- 172.21.15.59:0/1058892148 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ae40fd2f0 0x7f0ae410cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:24.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.790+0000 7f0ae9d9d640 1 --2- 172.21.15.59:0/1058892148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ae40fa240 0x7f0ae410c7a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:24.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.790+0000 7f0ae9d9d640 1 -- 172.21.15.59:0/1058892148 >> 172.21.15.59:0/1058892148 conn(0x7f0ae40f5d00 msgr2=0x7f0ae4100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:24.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.790+0000 7f0ae9d9d640 1 -- 172.21.15.59:0/1058892148 shutdown_connections 2024-01-23T08:50:24.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:24.790+0000 7f0ae9d9d640 1 -- 172.21.15.59:0/1058892148 wait complete. 2024-01-23T08:50:24.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:24 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/1589853922' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-23T08:50:24.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:24 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/3501941350' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-23T08:50:24.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:24 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/4279511794' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-01-23T08:50:24.798 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 19, 'max_mds': 2, 'flags': 50} 2024-01-23T08:50:24.798 INFO:teuthology.run_tasks:Running task kclient... 2024-01-23T08:50:24.822 INFO:tasks.kclient:Mounting kernel clients... 2024-01-23T08:50:24.822 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-01-23T08:50:24.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:24.822 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-01-23T08:50:24.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:24.837 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link delete ceph-brx 2024-01-23T08:50:24.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-23T08:50:24.909 INFO:teuthology.orchestra.run.smithi059.stderr:Cannot find device "ceph-brx" 2024-01-23T08:50:24.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:24.909 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-01-23T08:50:24.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:24.929 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link delete ceph-brx 2024-01-23T08:50:25.005 INFO:teuthology.orchestra.run.smithi137.stderr:Cannot find device "ceph-brx" 2024-01-23T08:50:25.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-23T08:50:25.007 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-01-23T08:50:25.007 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-23T08:50:25.007 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-23T08:50:25.007 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph fs ls 2024-01-23T08:50:25.303 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:50:25.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:25 smithi059 ceph-mon[28424]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-01-23T08:50:25.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:25 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/3651502973' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-01-23T08:50:25.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:25 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/1058892148' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-23T08:50:25.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:25 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:25.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:25 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:25.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:25 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:50:25.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:25 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:50:25.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:25 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:25.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:25 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:25.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:25 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:50:25.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:25 smithi137 ceph-mon[35787]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-01-23T08:50:25.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:25 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/3651502973' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-01-23T08:50:25.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:25 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/1058892148' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-23T08:50:25.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:25 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:25.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:25 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:25.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:25 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:50:25.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:25 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:50:25.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:25 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:25.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:25 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:50:25.794 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:25 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:50:26.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.817+0000 7fce4b7bb700 1 -- 172.21.15.59:0/1724397181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce440fe7c0 msgr2=0x7fce44106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:26.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.817+0000 7fce4b7bb700 1 --2- 172.21.15.59:0/1724397181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce440fe7c0 0x7fce44106e90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fce34004940 tx=0x7fce34009e30 comp rx=0 tx=0).stop 2024-01-23T08:50:26.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.817+0000 7fce4b7bb700 1 -- 172.21.15.59:0/1724397181 shutdown_connections 2024-01-23T08:50:26.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.817+0000 7fce4b7bb700 1 --2- 172.21.15.59:0/1724397181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce440fe7c0 0x7fce44106e90 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:26.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.817+0000 7fce4b7bb700 1 --2- 172.21.15.59:0/1724397181 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce440fde20 0x7fce440fe1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:26.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.817+0000 7fce4b7bb700 1 -- 172.21.15.59:0/1724397181 >> 172.21.15.59:0/1724397181 conn(0x7fce440f78a0 msgr2=0x7fce440f9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:26.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.817+0000 7fce4b7bb700 1 -- 172.21.15.59:0/1724397181 shutdown_connections 2024-01-23T08:50:26.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.817+0000 7fce4b7bb700 1 -- 172.21.15.59:0/1724397181 wait complete. 2024-01-23T08:50:26.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.817+0000 7fce4b7bb700 1 Processor -- start 2024-01-23T08:50:26.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.817+0000 7fce4b7bb700 1 -- start start 2024-01-23T08:50:26.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.818+0000 7fce4b7bb700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce440fde20 0x7fce441138d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:26.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.818+0000 7fce4b7bb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce440fe7c0 0x7fce44113e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.818+0000 7fce4b7bb700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce44114460 con 0x7fce440fe7c0 2024-01-23T08:50:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.818+0000 7fce4b7bb700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4410d790 con 0x7fce440fde20 2024-01-23T08:50:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.818+0000 7fce48d56700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce440fe7c0 0x7fce44113e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.818+0000 7fce48d56700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce440fe7c0 0x7fce44113e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48438/0 (socket says 172.21.15.59:48438) 2024-01-23T08:50:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.818+0000 7fce48d56700 1 -- 172.21.15.59:0/2116686184 learned_addr learned my addr 172.21.15.59:0/2116686184 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:50:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.818+0000 7fce49557700 1 --2- 172.21.15.59:0/2116686184 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce440fde20 0x7fce441138d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.818+0000 7fce48d56700 1 -- 172.21.15.59:0/2116686184 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce440fde20 msgr2=0x7fce441138d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.818+0000 7fce48d56700 1 --2- 172.21.15.59:0/2116686184 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce440fde20 0x7fce441138d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.818+0000 7fce48d56700 1 -- 172.21.15.59:0/2116686184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce34005040 con 0x7fce440fe7c0 2024-01-23T08:50:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.818+0000 7fce48d56700 1 --2- 172.21.15.59:0/2116686184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce440fe7c0 0x7fce44113e10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fce34009150 tx=0x7fce34009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.818+0000 7fce39ffb700 1 -- 172.21.15.59:0/2116686184 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce3400e040 con 0x7fce440fe7c0 2024-01-23T08:50:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.818+0000 7fce39ffb700 1 -- 172.21.15.59:0/2116686184 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce34004290 con 0x7fce440fe7c0 2024-01-23T08:50:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.818+0000 7fce39ffb700 1 -- 172.21.15.59:0/2116686184 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce3401c3f0 con 0x7fce440fe7c0 2024-01-23T08:50:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.818+0000 7fce4b7bb700 1 -- 172.21.15.59:0/2116686184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce4410da10 con 0x7fce440fe7c0 2024-01-23T08:50:26.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.819+0000 7fce4b7bb700 1 -- 172.21.15.59:0/2116686184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce4410df50 con 0x7fce440fe7c0 2024-01-23T08:50:26.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.820+0000 7fce4b7bb700 1 -- 172.21.15.59:0/2116686184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce44104550 con 0x7fce440fe7c0 2024-01-23T08:50:26.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.820+0000 7fce39ffb700 1 -- 172.21.15.59:0/2116686184 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fce34013420 con 0x7fce440fe7c0 2024-01-23T08:50:26.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.820+0000 7fce39ffb700 1 --2- 172.21.15.59:0/2116686184 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fce30068f60 0x7fce3006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:26.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.820+0000 7fce39ffb700 1 -- 172.21.15.59:0/2116686184 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fce34087f50 con 0x7fce440fe7c0 2024-01-23T08:50:26.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.820+0000 7fce49557700 1 --2- 172.21.15.59:0/2116686184 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fce30068f60 0x7fce3006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:26.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.821+0000 7fce49557700 1 --2- 172.21.15.59:0/2116686184 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fce30068f60 0x7fce3006b410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fce40004f40 tx=0x7fce4000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:26.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:26.825+0000 7fce39ffb700 1 -- 172.21.15.59:0/2116686184 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fce340b9a00 con 0x7fce440fe7c0 2024-01-23T08:50:27.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:27.059+0000 7fce4b7bb700 1 -- 172.21.15.59:0/2116686184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fce4410ef20 con 0x7fce440fe7c0 2024-01-23T08:50:27.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:27.059+0000 7fce39ffb700 1 -- 172.21.15.59:0/2116686184 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) v1 ==== 53+0+83 (secure 0 0 0) 0x7fce4410ef20 con 0x7fce440fe7c0 2024-01-23T08:50:27.056 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-01-23T08:50:27.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:27.061+0000 7fce4b7bb700 1 -- 172.21.15.59:0/2116686184 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fce30068f60 msgr2=0x7fce3006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:27.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:27.061+0000 7fce4b7bb700 1 --2- 172.21.15.59:0/2116686184 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fce30068f60 0x7fce3006b410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fce40004f40 tx=0x7fce4000b040 comp rx=0 tx=0).stop 2024-01-23T08:50:27.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:27.061+0000 7fce4b7bb700 1 -- 172.21.15.59:0/2116686184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce440fe7c0 msgr2=0x7fce44113e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:27.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:27.061+0000 7fce4b7bb700 1 --2- 172.21.15.59:0/2116686184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce440fe7c0 0x7fce44113e10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fce34009150 tx=0x7fce34009e30 comp rx=0 tx=0).stop 2024-01-23T08:50:27.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:27.061+0000 7fce4b7bb700 1 -- 172.21.15.59:0/2116686184 shutdown_connections 2024-01-23T08:50:27.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:27.061+0000 7fce4b7bb700 1 --2- 172.21.15.59:0/2116686184 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fce30068f60 0x7fce3006b410 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:27.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:27.061+0000 7fce4b7bb700 1 --2- 172.21.15.59:0/2116686184 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce440fde20 0x7fce441138d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:27.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:27.061+0000 7fce4b7bb700 1 --2- 172.21.15.59:0/2116686184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce440fe7c0 0x7fce44113e10 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:27.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:27.061+0000 7fce4b7bb700 1 -- 172.21.15.59:0/2116686184 >> 172.21.15.59:0/2116686184 conn(0x7fce440f78a0 msgr2=0x7fce441016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:27.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:27.061+0000 7fce4b7bb700 1 -- 172.21.15.59:0/2116686184 shutdown_connections 2024-01-23T08:50:27.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:27.061+0000 7fce4b7bb700 1 -- 172.21.15.59:0/2116686184 wait complete. 2024-01-23T08:50:27.643 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-23T08:50:27.643 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-23T08:50:27.643 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-01-23T08:50:27.643 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-23T08:50:27.643 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-23T08:50:27.643 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-23T08:50:27.643 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-23T08:50:27.643 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-23T08:50:27.643 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-23T08:50:27.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:27.644 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-01-23T08:50:27.659 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-23T08:50:27.659 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-23T08:50:27.660 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-01-23T08:50:27.660 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-01-23T08:50:27.660 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-01-23T08:50:27.660 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-01-23T08:50:27.660 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-23T08:50:27.660 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-01-23T08:50:27.660 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-01-23T08:50:27.660 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-23T08:50:27.660 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-01-23T08:50:27.660 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-01-23T08:50:27.660 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-23T08:50:27.660 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-01-23T08:50:27.660 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-01-23T08:50:27.660 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-23T08:50:27.661 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 41877sec preferred_lft 41877sec 2024-01-23T08:50:27.661 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-01-23T08:50:27.661 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-01-23T08:50:27.661 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-23T08:50:27.661 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-01-23T08:50:27.661 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-01-23T08:50:27.661 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-23T08:50:27.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:27.662 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-01-23T08:50:27.662 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-01-23T08:50:27.662 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link add name ceph-brx type bridge 2024-01-23T08:50:27.662 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip addr flush dev ceph-brx 2024-01-23T08:50:27.662 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set ceph-brx up 2024-01-23T08:50:27.662 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-23T08:50:27.662 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-01-23T08:50:27.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:27 smithi059 ceph-mon[28424]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.4 KiB/s rd, 0 B/s wr, 8 op/s 2024-01-23T08:50:27.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:27 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/2116686184' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-01-23T08:50:27.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:27 smithi137 ceph-mon[35787]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.4 KiB/s rd, 0 B/s wr, 8 op/s 2024-01-23T08:50:27.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:27 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/2116686184' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-01-23T08:50:27.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:27.874 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-23T08:50:27.941 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-01-23T08:50:27.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:27.943 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-01-23T08:50:28.012 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-01-23T08:50:28.012 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-23T08:50:28.012 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-23T08:50:28.012 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-23T08:50:28.012 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-23T08:50:28.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:28.013 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-01-23T08:50:28.013 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-01-23T08:50:28.013 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-23T08:50:28.013 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-23T08:50:28.013 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-23T08:50:28.013 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-01-23T08:50:28.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:28.281 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-01-23T08:50:28.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:28.295 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list-id 2024-01-23T08:50:28.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:28.351 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-01-23T08:50:28.351 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-01-23T08:50:28.351 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-23T08:50:28.351 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-01-23T08:50:28.351 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-01-23T08:50:28.434 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-01-23T08:50:28.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:28.434 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-01-23T08:50:28.434 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-01-23T08:50:28.434 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-01-23T08:50:28.434 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-23T08:50:28.434 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-01-23T08:50:28.434 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-01-23T08:50:28.434 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-01-23T08:50:28.435 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-01-23T08:50:28.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:28.571 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-01-23T08:50:28.571 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-01-23T08:50:28.571 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-01-23T08:50:28.571 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set dev brx.0 master ceph-brx 2024-01-23T08:50:28.571 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-01-23T08:50:28.613 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-23T08:50:28.613 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-23T08:50:28.667 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-23T08:50:28.667 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-23T08:50:28.668 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-23T08:50:28.723 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-23T08:50:28.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:28.724 DEBUG:teuthology.orchestra.run.smithi059:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-23T08:50:28.814 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-23T08:50:28.814 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-23T08:50:28.814 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-01-23T08:50:28.923 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-23T08:50:28.923 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-23T08:50:28.923 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-01-23T08:50:28.923 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.137:3300" 2024-01-23T08:50:28.923 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-23T08:50:28.933 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-23T08:50:28.933 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-23T08:50:28.933 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2024-01-23T08:50:28.933 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.137:3300" 2024-01-23T08:50:28.933 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-23T08:50:28.933 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.137:6789" 2024-01-23T08:50:28.933 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.137:6789:/ 2024-01-23T08:50:28.934 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=3fc07a38-b9cb-11ee-95b1-87774f69a715" will pass to kernel 2024-01-23T08:50:28.934 INFO:teuthology.orchestra.run.smithi059.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-01-23T08:50:28.934 INFO:teuthology.orchestra.run.smithi059.stdout: You just mounted a file system that supports labels which does not 2024-01-23T08:50:28.934 INFO:teuthology.orchestra.run.smithi059.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-01-23T08:50:28.934 INFO:teuthology.orchestra.run.smithi059.stdout: applications will generate AVC messages and not be allowed access to 2024-01-23T08:50:28.934 INFO:teuthology.orchestra.run.smithi059.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-01-23T08:50:28.936 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-23T08:50:28.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:28.936 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-23T08:50:28.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:28.967 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-23T08:50:29.026 INFO:teuthology.orchestra.run.smithi059.stdout:client24317 2024-01-23T08:50:29.027 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-23T08:50:29.027 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-23T08:50:29.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:29.083 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-23T08:50:29.144 INFO:teuthology.orchestra.run.smithi059.stdout:client24317 2024-01-23T08:50:29.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:29.144 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-23T08:50:29.204 INFO:teuthology.orchestra.run.smithi059.stdout:client24317 2024-01-23T08:50:29.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:29.204 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715.client24317/status') 2024-01-23T08:50:29.273 INFO:teuthology.orchestra.run.smithi059.stdout:instance: client.24317 (3)172.21.15.59:0/3140044931 2024-01-23T08:50:29.273 INFO:teuthology.orchestra.run.smithi059.stdout:blocklisted: false 2024-01-23T08:50:29.274 INFO:teuthology.orchestra.run.smithi059.stderr:0+1 records in 2024-01-23T08:50:29.274 INFO:teuthology.orchestra.run.smithi059.stderr:0+1 records out 2024-01-23T08:50:29.274 INFO:teuthology.orchestra.run.smithi059.stderr:71 bytes copied, 2.031e-05 s, 3.5 MB/s 2024-01-23T08:50:29.275 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-01-23T08:50:29.275 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-23T08:50:29.275 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-23T08:50:29.275 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -- ceph fs ls 2024-01-23T08:50:29.575 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:50:29.689 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:29 smithi059 ceph-mon[28424]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-01-23T08:50:29.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:29 smithi137 ceph-mon[35787]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-01-23T08:50:31.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.234+0000 7f89c70ce700 1 -- 172.21.15.59:0/4032505166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89c00fb400 msgr2=0x7f89c00fb850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:31.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.234+0000 7f89c70ce700 1 --2- 172.21.15.59:0/4032505166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89c00fb400 0x7f89c00fb850 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f89ac004940 tx=0x7f89ac009e30 comp rx=0 tx=0).stop 2024-01-23T08:50:31.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.234+0000 7f89c70ce700 1 -- 172.21.15.59:0/4032505166 shutdown_connections 2024-01-23T08:50:31.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.234+0000 7f89c70ce700 1 --2- 172.21.15.59:0/4032505166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89c00fb400 0x7f89c00fb850 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:31.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.234+0000 7f89c70ce700 1 --2- 172.21.15.59:0/4032505166 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89c0102fd0 0x7f89c01033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:31.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.234+0000 7f89c70ce700 1 -- 172.21.15.59:0/4032505166 >> 172.21.15.59:0/4032505166 conn(0x7f89c00713f0 msgr2=0x7f89c00717f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:31.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.234+0000 7f89c70ce700 1 -- 172.21.15.59:0/4032505166 shutdown_connections 2024-01-23T08:50:31.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.234+0000 7f89c70ce700 1 -- 172.21.15.59:0/4032505166 wait complete. 2024-01-23T08:50:31.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.234+0000 7f89c70ce700 1 Processor -- start 2024-01-23T08:50:31.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.234+0000 7f89c70ce700 1 -- start start 2024-01-23T08:50:31.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.235+0000 7f89c70ce700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89c00fb400 0x7f89c0113950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:31.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.235+0000 7f89c70ce700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89c0102fd0 0x7f89c0113e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:31.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.235+0000 7f89c70ce700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c0114570 con 0x7f89c0102fd0 2024-01-23T08:50:31.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.235+0000 7f89c70ce700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c010d810 con 0x7f89c00fb400 2024-01-23T08:50:31.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.235+0000 7f89bffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89c0102fd0 0x7f89c0113e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:31.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.235+0000 7f89bffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89c0102fd0 0x7f89c0113e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45886/0 (socket says 172.21.15.59:45886) 2024-01-23T08:50:31.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.235+0000 7f89bffff700 1 -- 172.21.15.59:0/2547226265 learned_addr learned my addr 172.21.15.59:0/2547226265 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:50:31.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.235+0000 7f89bffff700 1 -- 172.21.15.59:0/2547226265 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89c00fb400 msgr2=0x7f89c0113950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:31.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.235+0000 7f89c4e6a700 1 --2- 172.21.15.59:0/2547226265 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89c00fb400 0x7f89c0113950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:31.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.235+0000 7f89bffff700 1 --2- 172.21.15.59:0/2547226265 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89c00fb400 0x7f89c0113950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:31.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.235+0000 7f89bffff700 1 -- 172.21.15.59:0/2547226265 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ac005040 con 0x7f89c0102fd0 2024-01-23T08:50:31.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.235+0000 7f89c4e6a700 1 --2- 172.21.15.59:0/2547226265 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89c00fb400 0x7f89c0113950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-23T08:50:31.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.235+0000 7f89bffff700 1 --2- 172.21.15.59:0/2547226265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89c0102fd0 0x7f89c0113e90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f89ac009fd0 tx=0x7f89ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:31.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.235+0000 7f89bd7fa700 1 -- 172.21.15.59:0/2547226265 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f89ac00e040 con 0x7f89c0102fd0 2024-01-23T08:50:31.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.235+0000 7f89bd7fa700 1 -- 172.21.15.59:0/2547226265 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f89ac00bc50 con 0x7f89c0102fd0 2024-01-23T08:50:31.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.235+0000 7f89c70ce700 1 -- 172.21.15.59:0/2547226265 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89c010da90 con 0x7f89c0102fd0 2024-01-23T08:50:31.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.235+0000 7f89bd7fa700 1 -- 172.21.15.59:0/2547226265 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f89ac013ea0 con 0x7f89c0102fd0 2024-01-23T08:50:31.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.236+0000 7f89c70ce700 1 -- 172.21.15.59:0/2547226265 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89c010dfd0 con 0x7f89c0102fd0 2024-01-23T08:50:31.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.236+0000 7f89c70ce700 1 -- 172.21.15.59:0/2547226265 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89c0072f10 con 0x7f89c0102fd0 2024-01-23T08:50:31.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.237+0000 7f89bd7fa700 1 -- 172.21.15.59:0/2547226265 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f89ac013420 con 0x7f89c0102fd0 2024-01-23T08:50:31.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.237+0000 7f89bd7fa700 1 --2- 172.21.15.59:0/2547226265 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f89b0068f70 0x7f89b006b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:31.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.237+0000 7f89bd7fa700 1 -- 172.21.15.59:0/2547226265 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f89ac0882e0 con 0x7f89c0102fd0 2024-01-23T08:50:31.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.237+0000 7f89c4e6a700 1 --2- 172.21.15.59:0/2547226265 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f89b0068f70 0x7f89b006b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:31.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.238+0000 7f89c4e6a700 1 --2- 172.21.15.59:0/2547226265 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f89b0068f70 0x7f89b006b420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f89b4006fd0 tx=0x7f89b4008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:31.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.242+0000 7f89bd7fa700 1 -- 172.21.15.59:0/2547226265 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f89ac058150 con 0x7f89c0102fd0 2024-01-23T08:50:31.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:31 smithi059 ceph-mon[28424]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-01-23T08:50:31.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.478+0000 7f89c70ce700 1 -- 172.21.15.59:0/2547226265 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f89c004bd60 con 0x7f89c0102fd0 2024-01-23T08:50:31.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.479+0000 7f89bd7fa700 1 -- 172.21.15.59:0/2547226265 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) v1 ==== 53+0+83 (secure 0 0 0) 0x7f89ac057ce0 con 0x7f89c0102fd0 2024-01-23T08:50:31.476 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-01-23T08:50:31.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.481+0000 7f89c70ce700 1 -- 172.21.15.59:0/2547226265 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f89b0068f70 msgr2=0x7f89b006b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:31.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.481+0000 7f89c70ce700 1 --2- 172.21.15.59:0/2547226265 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f89b0068f70 0x7f89b006b420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f89b4006fd0 tx=0x7f89b4008040 comp rx=0 tx=0).stop 2024-01-23T08:50:31.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.481+0000 7f89c70ce700 1 -- 172.21.15.59:0/2547226265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89c0102fd0 msgr2=0x7f89c0113e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:31.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.481+0000 7f89c70ce700 1 --2- 172.21.15.59:0/2547226265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89c0102fd0 0x7f89c0113e90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f89ac009fd0 tx=0x7f89ac009e30 comp rx=0 tx=0).stop 2024-01-23T08:50:31.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.481+0000 7f89c70ce700 1 -- 172.21.15.59:0/2547226265 shutdown_connections 2024-01-23T08:50:31.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.481+0000 7f89c70ce700 1 --2- 172.21.15.59:0/2547226265 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f89b0068f70 0x7f89b006b420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:31.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.481+0000 7f89c70ce700 1 --2- 172.21.15.59:0/2547226265 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89c00fb400 0x7f89c0113950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:31.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.481+0000 7f89c70ce700 1 --2- 172.21.15.59:0/2547226265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89c0102fd0 0x7f89c0113e90 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:31.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.481+0000 7f89c70ce700 1 -- 172.21.15.59:0/2547226265 >> 172.21.15.59:0/2547226265 conn(0x7f89c00713f0 msgr2=0x7f89c00f9550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:31.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.481+0000 7f89c70ce700 1 -- 172.21.15.59:0/2547226265 shutdown_connections 2024-01-23T08:50:31.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:31.481+0000 7f89c70ce700 1 -- 172.21.15.59:0/2547226265 wait complete. 2024-01-23T08:50:31.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:31 smithi137 ceph-mon[35787]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-01-23T08:50:32.097 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-23T08:50:32.097 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-23T08:50:32.097 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi137.front.sepia.ceph.com 2024-01-23T08:50:32.097 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-23T08:50:32.097 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-23T08:50:32.097 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-23T08:50:32.097 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-23T08:50:32.097 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-23T08:50:32.097 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-23T08:50:32.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:32.098 DEBUG:teuthology.orchestra.run.smithi137:> ip addr 2024-01-23T08:50:32.117 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-23T08:50:32.117 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-23T08:50:32.117 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2024-01-23T08:50:32.117 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-23T08:50:32.117 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2024-01-23T08:50:32.117 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-23T08:50:32.118 INFO:teuthology.orchestra.run.smithi137.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-23T08:50:32.118 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7e brd ff:ff:ff:ff:ff:ff 2024-01-23T08:50:32.118 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f0 2024-01-23T08:50:32.118 INFO:teuthology.orchestra.run.smithi137.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-23T08:50:32.118 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7f brd ff:ff:ff:ff:ff:ff 2024-01-23T08:50:32.118 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f1 2024-01-23T08:50:32.118 INFO:teuthology.orchestra.run.smithi137.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-23T08:50:32.118 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:36 brd ff:ff:ff:ff:ff:ff 2024-01-23T08:50:32.118 INFO:teuthology.orchestra.run.smithi137.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-23T08:50:32.118 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:37 brd ff:ff:ff:ff:ff:ff 2024-01-23T08:50:32.119 INFO:teuthology.orchestra.run.smithi137.stdout: inet 172.21.15.137/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-23T08:50:32.119 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft 41867sec preferred_lft 41867sec 2024-01-23T08:50:32.119 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::ec4:7aff:fe88:8137/64 scope link 2024-01-23T08:50:32.119 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-23T08:50:32.119 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-23T08:50:32.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:32.119 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-23T08:50:32.119 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-23T08:50:32.119 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link add name ceph-brx type bridge 2024-01-23T08:50:32.119 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip addr flush dev ceph-brx 2024-01-23T08:50:32.120 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set ceph-brx up 2024-01-23T08:50:32.120 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-23T08:50:32.120 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-23T08:50:32.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:32.301 DEBUG:teuthology.orchestra.run.smithi137:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-23T08:50:32.321 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:32 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/2547226265' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-01-23T08:50:32.328 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-01-23T08:50:32.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:32.330 DEBUG:teuthology.orchestra.run.smithi137:> route 2024-01-23T08:50:32.398 INFO:teuthology.orchestra.run.smithi137.stdout:Kernel IP routing table 2024-01-23T08:50:32.398 INFO:teuthology.orchestra.run.smithi137.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-23T08:50:32.398 INFO:teuthology.orchestra.run.smithi137.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-23T08:50:32.398 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-23T08:50:32.398 INFO:teuthology.orchestra.run.smithi137.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-23T08:50:32.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:32.399 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-23T08:50:32.399 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-23T08:50:32.399 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-23T08:50:32.399 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-23T08:50:32.399 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-23T08:50:32.399 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-23T08:50:32.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:32 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/2547226265' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-01-23T08:50:32.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:32.707 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-01-23T08:50:32.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:32.722 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list-id 2024-01-23T08:50:32.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:32.777 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-23T08:50:32.777 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-23T08:50:32.777 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-23T08:50:32.777 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-01-23T08:50:32.777 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-23T08:50:32.860 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-01-23T08:50:32.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:32.861 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-23T08:50:32.861 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-23T08:50:32.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-01-23T08:50:32.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-23T08:50:32.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-01-23T08:50:32.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-01-23T08:50:32.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-01-23T08:50:32.861 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-23T08:50:32.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:32.998 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-23T08:50:32.998 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-23T08:50:32.998 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set brx.0 up 2024-01-23T08:50:32.998 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set dev brx.0 master ceph-brx 2024-01-23T08:50:32.998 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-23T08:50:33.039 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-23T08:50:33.040 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-23T08:50:33.095 INFO:teuthology.orchestra.run.smithi137.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-23T08:50:33.095 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-23T08:50:33.095 DEBUG:teuthology.orchestra.run.smithi137:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-23T08:50:33.150 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-23T08:50:33.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:33.150 DEBUG:teuthology.orchestra.run.smithi137:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-23T08:50:33.213 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:33 smithi137 ceph-mon[35787]: pgmap v196: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-01-23T08:50:33.249 INFO:teuthology.orchestra.run.smithi137.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-23T08:50:33.249 INFO:teuthology.orchestra.run.smithi137.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-01-23T08:50:33.249 INFO:teuthology.orchestra.run.smithi137.stdout:invalid new device string format 2024-01-23T08:50:33.356 INFO:teuthology.orchestra.run.smithi137.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-23T08:50:33.356 INFO:teuthology.orchestra.run.smithi137.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-01-23T08:50:33.356 INFO:teuthology.orchestra.run.smithi137.stdout:invalid new device string format 2024-01-23T08:50:33.356 INFO:teuthology.orchestra.run.smithi137.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.137:3300" 2024-01-23T08:50:33.356 INFO:teuthology.orchestra.run.smithi137.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-23T08:50:33.360 INFO:teuthology.orchestra.run.smithi137.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-23T08:50:33.360 INFO:teuthology.orchestra.run.smithi137.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-01-23T08:50:33.360 INFO:teuthology.orchestra.run.smithi137.stdout:invalid new device string format 2024-01-23T08:50:33.360 INFO:teuthology.orchestra.run.smithi137.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.137:3300" 2024-01-23T08:50:33.360 INFO:teuthology.orchestra.run.smithi137.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-23T08:50:33.360 INFO:teuthology.orchestra.run.smithi137.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.137:6789" 2024-01-23T08:50:33.361 INFO:teuthology.orchestra.run.smithi137.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.137:6789:/ 2024-01-23T08:50:33.361 INFO:teuthology.orchestra.run.smithi137.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=3fc07a38-b9cb-11ee-95b1-87774f69a715" will pass to kernel 2024-01-23T08:50:33.361 INFO:teuthology.orchestra.run.smithi137.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-01-23T08:50:33.361 INFO:teuthology.orchestra.run.smithi137.stdout: You just mounted a file system that supports labels which does not 2024-01-23T08:50:33.361 INFO:teuthology.orchestra.run.smithi137.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-01-23T08:50:33.361 INFO:teuthology.orchestra.run.smithi137.stdout: applications will generate AVC messages and not be allowed access to 2024-01-23T08:50:33.361 INFO:teuthology.orchestra.run.smithi137.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-01-23T08:50:33.363 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-23T08:50:33.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:33.363 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-23T08:50:33.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:33.431 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-23T08:50:33.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:33 smithi059 ceph-mon[28424]: pgmap v196: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-01-23T08:50:33.487 INFO:teuthology.orchestra.run.smithi137.stdout:client24319 2024-01-23T08:50:33.488 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-23T08:50:33.488 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-23T08:50:33.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:33.544 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-23T08:50:33.600 INFO:teuthology.orchestra.run.smithi137.stdout:client24319 2024-01-23T08:50:33.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:33.600 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-23T08:50:33.656 INFO:teuthology.orchestra.run.smithi137.stdout:client24319 2024-01-23T08:50:33.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T08:50:33.656 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715.client24319/status') 2024-01-23T08:50:33.723 INFO:teuthology.orchestra.run.smithi137.stdout:instance: client.24319 (3)192.168.0.1:0/3990898158 2024-01-23T08:50:33.723 INFO:teuthology.orchestra.run.smithi137.stdout:blocklisted: false 2024-01-23T08:50:33.724 INFO:teuthology.orchestra.run.smithi137.stderr:0+1 records in 2024-01-23T08:50:33.724 INFO:teuthology.orchestra.run.smithi137.stderr:0+1 records out 2024-01-23T08:50:33.724 INFO:teuthology.orchestra.run.smithi137.stderr:70 bytes copied, 2.2972e-05 s, 3.0 MB/s 2024-01-23T08:50:33.725 INFO:teuthology.run_tasks:Running task print... 2024-01-23T08:50:33.735 INFO:teuthology.task.print:**** done client 2024-01-23T08:50:33.735 INFO:teuthology.run_tasks:Running task parallel... 2024-01-23T08:50:33.780 INFO:teuthology.task.parallel:starting parallel... 2024-01-23T08:50:33.780 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-01-23T08:50:33.781 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-01-23T08:50:33.781 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2024-01-23T08:50:33.781 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-01-23T08:50:33.782 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-01-23T08:50:33.782 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-01-23T08:50:33.786 INFO:tasks.workunit:Pulling workunits from ref 9db6bac5568c8be6cfd98da20bd2b62582f0776f 2024-01-23T08:50:33.786 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-01-23T08:50:33.786 INFO:tasks.workunit:timeout=3h 2024-01-23T08:50:33.786 INFO:tasks.workunit:cleanup=True 2024-01-23T08:50:33.786 DEBUG:teuthology.orchestra.run.smithi059:> stat -- /home/ubuntu/cephtest/mnt.0 2024-01-23T08:50:33.806 INFO:teuthology.orchestra.run.smithi059.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-01-23T08:50:33.806 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-01-23T08:50:33.806 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 74h/116d Inode: 1 Links: 2 2024-01-23T08:50:33.806 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-23T08:50:33.806 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-01-23T08:50:33.807 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-01-23T08:50:33.807 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-01-23 08:50:02.055091658 +0000 2024-01-23T08:50:33.807 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-01-23 08:50:28.964205107 +0000 2024-01-23T08:50:33.807 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-01-23 08:50:02.055091658 +0000 2024-01-23T08:50:33.807 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-01-23T08:50:33.807 DEBUG:teuthology.orchestra.run.smithi059:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-01-23T08:50:33.880 DEBUG:teuthology.orchestra.run.smithi137:> stat -- /home/ubuntu/cephtest/mnt.1 2024-01-23T08:50:33.897 INFO:teuthology.orchestra.run.smithi137.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-01-23T08:50:33.897 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-01-23T08:50:33.897 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 73h/115d Inode: 1 Links: 3 2024-01-23T08:50:33.897 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-23T08:50:33.897 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-01-23T08:50:33.897 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-01-23T08:50:33.897 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-01-23 08:50:33.877141894 +0000 2024-01-23T08:50:33.897 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-01-23 08:50:33.877141894 +0000 2024-01-23T08:50:33.897 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-01-23 08:50:02.055091658 +0000 2024-01-23T08:50:33.897 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-01-23T08:50:33.898 DEBUG:teuthology.orchestra.run.smithi137:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-01-23T08:50:33.979 DEBUG:teuthology.orchestra.run.smithi059:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/cbodley/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 9db6bac5568c8be6cfd98da20bd2b62582f0776f 2024-01-23T08:50:33.979 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/cbodley/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 9db6bac5568c8be6cfd98da20bd2b62582f0776f 2024-01-23T08:50:33.995 INFO:tasks.workunit.client.0.smithi059.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-01-23T08:50:34.036 INFO:tasks.workunit.client.1.smithi137.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-01-23T08:50:34.066 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:50:35.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:35 smithi059 ceph-mon[28424]: pgmap v197: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-01-23T08:50:35.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:35 smithi137 ceph-mon[35787]: pgmap v197: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-01-23T08:50:35.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.863+0000 7ff584ac4700 1 -- 172.21.15.59:0/2653856869 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff580066270 msgr2=0x7ff580101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:35.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.863+0000 7ff584ac4700 1 --2- 172.21.15.59:0/2653856869 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff580066270 0x7ff580101d80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff570004940 tx=0x7ff570009e30 comp rx=0 tx=0).stop 2024-01-23T08:50:35.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.863+0000 7ff584ac4700 1 -- 172.21.15.59:0/2653856869 shutdown_connections 2024-01-23T08:50:35.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.863+0000 7ff584ac4700 1 --2- 172.21.15.59:0/2653856869 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff5801022c0 0x7ff5801046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:35.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.863+0000 7ff584ac4700 1 --2- 172.21.15.59:0/2653856869 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff580066270 0x7ff580101d80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:35.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.863+0000 7ff584ac4700 1 -- 172.21.15.59:0/2653856869 >> 172.21.15.59:0/2653856869 conn(0x7ff5800f7370 msgr2=0x7ff5800f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:35.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.863+0000 7ff584ac4700 1 -- 172.21.15.59:0/2653856869 shutdown_connections 2024-01-23T08:50:35.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.863+0000 7ff584ac4700 1 -- 172.21.15.59:0/2653856869 wait complete. 2024-01-23T08:50:35.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.864+0000 7ff584ac4700 1 Processor -- start 2024-01-23T08:50:35.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.864+0000 7ff584ac4700 1 -- start start 2024-01-23T08:50:35.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.864+0000 7ff584ac4700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff580066270 0x7ff58010f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:35.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.864+0000 7ff584ac4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5801022c0 0x7ff58010fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:35.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.864+0000 7ff584ac4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff580110500 con 0x7ff5801022c0 2024-01-23T08:50:35.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.864+0000 7ff584ac4700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff580110d40 con 0x7ff580066270 2024-01-23T08:50:35.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.864+0000 7ff57dd9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5801022c0 0x7ff58010fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:35.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.864+0000 7ff57dd9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5801022c0 0x7ff58010fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45910/0 (socket says 172.21.15.59:45910) 2024-01-23T08:50:35.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.864+0000 7ff57dd9b700 1 -- 172.21.15.59:0/105304961 learned_addr learned my addr 172.21.15.59:0/105304961 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:50:35.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.864+0000 7ff57e59c700 1 --2- 172.21.15.59:0/105304961 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff580066270 0x7ff58010f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:35.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.865+0000 7ff57dd9b700 1 -- 172.21.15.59:0/105304961 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff580066270 msgr2=0x7ff58010f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:35.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.865+0000 7ff57dd9b700 1 --2- 172.21.15.59:0/105304961 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff580066270 0x7ff58010f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:35.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.865+0000 7ff57dd9b700 1 -- 172.21.15.59:0/105304961 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff570005040 con 0x7ff5801022c0 2024-01-23T08:50:35.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.865+0000 7ff57dd9b700 1 --2- 172.21.15.59:0/105304961 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5801022c0 0x7ff58010fe70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff57400b790 tx=0x7ff574009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:35.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.865+0000 7ff56effd700 1 -- 172.21.15.59:0/105304961 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff57400f6f0 con 0x7ff5801022c0 2024-01-23T08:50:35.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.865+0000 7ff56effd700 1 -- 172.21.15.59:0/105304961 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff57400fd00 con 0x7ff5801022c0 2024-01-23T08:50:35.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.865+0000 7ff584ac4700 1 -- 172.21.15.59:0/105304961 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff580111020 con 0x7ff5801022c0 2024-01-23T08:50:35.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.865+0000 7ff56effd700 1 -- 172.21.15.59:0/105304961 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5740103a0 con 0x7ff5801022c0 2024-01-23T08:50:35.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.865+0000 7ff584ac4700 1 -- 172.21.15.59:0/105304961 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff580111560 con 0x7ff5801022c0 2024-01-23T08:50:35.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.866+0000 7ff56cff9700 1 -- 172.21.15.59:0/105304961 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5640052b0 con 0x7ff5801022c0 2024-01-23T08:50:35.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.867+0000 7ff56effd700 1 -- 172.21.15.59:0/105304961 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7ff57400f850 con 0x7ff5801022c0 2024-01-23T08:50:35.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.868+0000 7ff56effd700 1 --2- 172.21.15.59:0/105304961 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7ff568068f60 0x7ff56806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:35.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.868+0000 7ff56effd700 1 -- 172.21.15.59:0/105304961 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff574086c00 con 0x7ff5801022c0 2024-01-23T08:50:35.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.868+0000 7ff57e59c700 1 --2- 172.21.15.59:0/105304961 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7ff568068f60 0x7ff56806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:35.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.868+0000 7ff57e59c700 1 --2- 172.21.15.59:0/105304961 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7ff568068f60 0x7ff56806b410 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff57000b3b0 tx=0x7ff570009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:35.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:35.872+0000 7ff56effd700 1 -- 172.21.15.59:0/105304961 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ff574056ab0 con 0x7ff5801022c0 2024-01-23T08:50:36.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:36.098+0000 7ff56cff9700 1 -- 172.21.15.59:0/105304961 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7ff564005560 con 0x7ff5801022c0 2024-01-23T08:50:36.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:36.098+0000 7ff56effd700 1 -- 172.21.15.59:0/105304961 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7ff574056640 con 0x7ff5801022c0 2024-01-23T08:50:36.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:36.101+0000 7ff56cff9700 1 -- 172.21.15.59:0/105304961 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7ff568068f60 msgr2=0x7ff56806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:36.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:36.101+0000 7ff56cff9700 1 --2- 172.21.15.59:0/105304961 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7ff568068f60 0x7ff56806b410 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff57000b3b0 tx=0x7ff570009e30 comp rx=0 tx=0).stop 2024-01-23T08:50:36.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:36.101+0000 7ff56cff9700 1 -- 172.21.15.59:0/105304961 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5801022c0 msgr2=0x7ff58010fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:36.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:36.101+0000 7ff56cff9700 1 --2- 172.21.15.59:0/105304961 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5801022c0 0x7ff58010fe70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff57400b790 tx=0x7ff574009220 comp rx=0 tx=0).stop 2024-01-23T08:50:36.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:36.101+0000 7ff56cff9700 1 -- 172.21.15.59:0/105304961 shutdown_connections 2024-01-23T08:50:36.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:36.101+0000 7ff56cff9700 1 --2- 172.21.15.59:0/105304961 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7ff568068f60 0x7ff56806b410 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:36.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:36.101+0000 7ff56cff9700 1 --2- 172.21.15.59:0/105304961 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff580066270 0x7ff58010f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:36.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:36.101+0000 7ff56cff9700 1 --2- 172.21.15.59:0/105304961 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5801022c0 0x7ff58010fe70 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:36.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:36.101+0000 7ff56cff9700 1 -- 172.21.15.59:0/105304961 >> 172.21.15.59:0/105304961 conn(0x7ff5800f7370 msgr2=0x7ff5800f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:36.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:36.101+0000 7ff56cff9700 1 -- 172.21.15.59:0/105304961 shutdown_connections 2024-01-23T08:50:36.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:36.101+0000 7ff56cff9700 1 -- 172.21.15.59:0/105304961 wait complete. 2024-01-23T08:50:36.749 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-01-23T08:50:37.056 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:50:37.432 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:37 smithi059 ceph-mon[28424]: pgmap v198: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 938 B/s wr, 5 op/s 2024-01-23T08:50:37.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:37 smithi137 ceph-mon[35787]: pgmap v198: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 938 B/s wr, 5 op/s 2024-01-23T08:50:38.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.806+0000 7f32f8ad1700 1 -- 172.21.15.59:0/3654772309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f4066b00 msgr2=0x7f32f40ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:38.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.806+0000 7f32f8ad1700 1 --2- 172.21.15.59:0/3654772309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f4066b00 0x7f32f40ff980 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f32e4004940 tx=0x7f32e4009e30 comp rx=0 tx=0).stop 2024-01-23T08:50:38.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.806+0000 7f32f8ad1700 1 -- 172.21.15.59:0/3654772309 shutdown_connections 2024-01-23T08:50:38.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.806+0000 7f32f8ad1700 1 --2- 172.21.15.59:0/3654772309 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32f40fff50 0x7f32f41023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:38.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.806+0000 7f32f8ad1700 1 --2- 172.21.15.59:0/3654772309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f4066b00 0x7f32f40ff980 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:38.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.806+0000 7f32f8ad1700 1 -- 172.21.15.59:0/3654772309 >> 172.21.15.59:0/3654772309 conn(0x7f32f40f7390 msgr2=0x7f32f40f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:38.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.806+0000 7f32f8ad1700 1 -- 172.21.15.59:0/3654772309 shutdown_connections 2024-01-23T08:50:38.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.806+0000 7f32f8ad1700 1 -- 172.21.15.59:0/3654772309 wait complete. 2024-01-23T08:50:38.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.807+0000 7f32f8ad1700 1 Processor -- start 2024-01-23T08:50:38.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.807+0000 7f32f8ad1700 1 -- start start 2024-01-23T08:50:38.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.807+0000 7f32f8ad1700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32f4066b00 0x7f32f410d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:38.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.807+0000 7f32f8ad1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f40fff50 0x7f32f410dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:38.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.807+0000 7f32f8ad1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f410e380 con 0x7f32f40fff50 2024-01-23T08:50:38.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.807+0000 7f32f8ad1700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f410ec20 con 0x7f32f4066b00 2024-01-23T08:50:38.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.807+0000 7f32f1d9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f40fff50 0x7f32f410dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:38.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.807+0000 7f32f1d9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f40fff50 0x7f32f410dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41166/0 (socket says 172.21.15.59:41166) 2024-01-23T08:50:38.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.807+0000 7f32f1d9b700 1 -- 172.21.15.59:0/1246012734 learned_addr learned my addr 172.21.15.59:0/1246012734 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:50:38.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.807+0000 7f32f259c700 1 --2- 172.21.15.59:0/1246012734 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32f4066b00 0x7f32f410d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:38.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.807+0000 7f32f1d9b700 1 -- 172.21.15.59:0/1246012734 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32f4066b00 msgr2=0x7f32f410d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:38.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.807+0000 7f32f1d9b700 1 --2- 172.21.15.59:0/1246012734 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32f4066b00 0x7f32f410d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:38.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.807+0000 7f32f1d9b700 1 -- 172.21.15.59:0/1246012734 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e4005040 con 0x7f32f40fff50 2024-01-23T08:50:38.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.807+0000 7f32f259c700 1 --2- 172.21.15.59:0/1246012734 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32f4066b00 0x7f32f410d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-23T08:50:38.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.808+0000 7f32f1d9b700 1 --2- 172.21.15.59:0/1246012734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f40fff50 0x7f32f410dd80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f32e800eb40 tx=0x7f32e800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:38.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.808+0000 7f32e2ffd700 1 -- 172.21.15.59:0/1246012734 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32e8007cc0 con 0x7f32f40fff50 2024-01-23T08:50:38.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.808+0000 7f32e2ffd700 1 -- 172.21.15.59:0/1246012734 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f32e800f040 con 0x7f32f40fff50 2024-01-23T08:50:38.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.808+0000 7f32f8ad1700 1 -- 172.21.15.59:0/1246012734 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f410eea0 con 0x7f32f40fff50 2024-01-23T08:50:38.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.808+0000 7f32e2ffd700 1 -- 172.21.15.59:0/1246012734 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32e800c8f0 con 0x7f32f40fff50 2024-01-23T08:50:38.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.808+0000 7f32f8ad1700 1 -- 172.21.15.59:0/1246012734 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f410f3e0 con 0x7f32f40fff50 2024-01-23T08:50:38.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.809+0000 7f32f8ad1700 1 -- 172.21.15.59:0/1246012734 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f41076f0 con 0x7f32f40fff50 2024-01-23T08:50:38.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.809+0000 7f32e2ffd700 1 -- 172.21.15.59:0/1246012734 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f32e800cb60 con 0x7f32f40fff50 2024-01-23T08:50:38.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.810+0000 7f32e2ffd700 1 --2- 172.21.15.59:0/1246012734 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f32dc068f60 0x7f32dc06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:38.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.810+0000 7f32e2ffd700 1 -- 172.21.15.59:0/1246012734 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f32e8086820 con 0x7f32f40fff50 2024-01-23T08:50:38.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.810+0000 7f32f259c700 1 --2- 172.21.15.59:0/1246012734 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f32dc068f60 0x7f32dc06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:38.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.810+0000 7f32f259c700 1 --2- 172.21.15.59:0/1246012734 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f32dc068f60 0x7f32dc06b410 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f32e4000c00 tx=0x7f32e401c040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:38.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:38.814+0000 7f32e2ffd700 1 -- 172.21.15.59:0/1246012734 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f32e804da00 con 0x7f32f40fff50 2024-01-23T08:50:39.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:39.008+0000 7f32f8ad1700 1 -- 172.21.15.59:0/1246012734 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f32f4105900 con 0x7f32f40fff50 2024-01-23T08:50:39.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:39.008+0000 7f32e2ffd700 1 -- 172.21.15.59:0/1246012734 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f32e8056760 con 0x7f32f40fff50 2024-01-23T08:50:39.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:39.010+0000 7f32f8ad1700 1 -- 172.21.15.59:0/1246012734 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f32dc068f60 msgr2=0x7f32dc06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:39.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:39.010+0000 7f32f8ad1700 1 --2- 172.21.15.59:0/1246012734 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f32dc068f60 0x7f32dc06b410 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f32e4000c00 tx=0x7f32e401c040 comp rx=0 tx=0).stop 2024-01-23T08:50:39.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:39.010+0000 7f32f8ad1700 1 -- 172.21.15.59:0/1246012734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f40fff50 msgr2=0x7f32f410dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:39.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:39.010+0000 7f32f8ad1700 1 --2- 172.21.15.59:0/1246012734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f40fff50 0x7f32f410dd80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f32e800eb40 tx=0x7f32e800b040 comp rx=0 tx=0).stop 2024-01-23T08:50:39.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:39.010+0000 7f32f8ad1700 1 -- 172.21.15.59:0/1246012734 shutdown_connections 2024-01-23T08:50:39.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:39.010+0000 7f32f8ad1700 1 --2- 172.21.15.59:0/1246012734 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f32dc068f60 0x7f32dc06b410 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:39.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:39.010+0000 7f32f8ad1700 1 --2- 172.21.15.59:0/1246012734 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32f4066b00 0x7f32f410d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:39.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:39.010+0000 7f32f8ad1700 1 --2- 172.21.15.59:0/1246012734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f40fff50 0x7f32f410dd80 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:39.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:39.010+0000 7f32f8ad1700 1 -- 172.21.15.59:0/1246012734 >> 172.21.15.59:0/1246012734 conn(0x7f32f40f7390 msgr2=0x7f32f40f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:39.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:39.011+0000 7f32f8ad1700 1 -- 172.21.15.59:0/1246012734 shutdown_connections 2024-01-23T08:50:39.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:39.011+0000 7f32f8ad1700 1 -- 172.21.15.59:0/1246012734 wait complete. 2024-01-23T08:50:39.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:39 smithi059 ceph-mon[28424]: pgmap v199: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 938 B/s wr, 4 op/s 2024-01-23T08:50:39.675 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c 'ceph config set global log_to_journald false --force' 2024-01-23T08:50:39.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:39 smithi137 ceph-mon[35787]: pgmap v199: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 938 B/s wr, 4 op/s 2024-01-23T08:50:39.943 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:50:41.132 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:41 smithi059 ceph-mon[28424]: pgmap v200: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 938 B/s wr, 4 op/s 2024-01-23T08:50:41.132 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:41 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:50:41.542 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:41 smithi137 ceph-mon[35787]: pgmap v200: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 938 B/s wr, 4 op/s 2024-01-23T08:50:41.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:41 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:50:41.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.712+0000 7fb665d71700 1 -- 172.21.15.59:0/1850890681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6600fff50 msgr2=0x7fb6601023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:41.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.712+0000 7fb665d71700 1 --2- 172.21.15.59:0/1850890681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6600fff50 0x7fb6601023b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb654004940 tx=0x7fb654009e30 comp rx=0 tx=0).stop 2024-01-23T08:50:41.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.713+0000 7fb665d71700 1 -- 172.21.15.59:0/1850890681 shutdown_connections 2024-01-23T08:50:41.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.713+0000 7fb665d71700 1 --2- 172.21.15.59:0/1850890681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6600fff50 0x7fb6601023b0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:41.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.713+0000 7fb665d71700 1 --2- 172.21.15.59:0/1850890681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb660066b00 0x7fb6600ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:41.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.713+0000 7fb665d71700 1 -- 172.21.15.59:0/1850890681 >> 172.21.15.59:0/1850890681 conn(0x7fb6600f7390 msgr2=0x7fb6600f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:41.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.713+0000 7fb665d71700 1 -- 172.21.15.59:0/1850890681 shutdown_connections 2024-01-23T08:50:41.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.713+0000 7fb665d71700 1 -- 172.21.15.59:0/1850890681 wait complete. 2024-01-23T08:50:41.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.713+0000 7fb665d71700 1 Processor -- start 2024-01-23T08:50:41.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.713+0000 7fb665d71700 1 -- start start 2024-01-23T08:50:41.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.714+0000 7fb665d71700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb660066b00 0x7fb66010d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:41.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.714+0000 7fb665d71700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6600fff50 0x7fb66010dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:41.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.714+0000 7fb665d71700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66010e390 con 0x7fb6600fff50 2024-01-23T08:50:41.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.714+0000 7fb665d71700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66010ec30 con 0x7fb660066b00 2024-01-23T08:50:41.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.714+0000 7fb65effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6600fff50 0x7fb66010dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:41.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.714+0000 7fb65effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6600fff50 0x7fb66010dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41182/0 (socket says 172.21.15.59:41182) 2024-01-23T08:50:41.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.714+0000 7fb65f7fe700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb660066b00 0x7fb66010d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:41.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.714+0000 7fb65effd700 1 -- 172.21.15.59:0/16876983 learned_addr learned my addr 172.21.15.59:0/16876983 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:50:41.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.714+0000 7fb65effd700 1 -- 172.21.15.59:0/16876983 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb660066b00 msgr2=0x7fb66010d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:41.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.714+0000 7fb65effd700 1 --2- 172.21.15.59:0/16876983 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb660066b00 0x7fb66010d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:41.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.714+0000 7fb65effd700 1 -- 172.21.15.59:0/16876983 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb654005040 con 0x7fb6600fff50 2024-01-23T08:50:41.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.714+0000 7fb65f7fe700 1 --2- 172.21.15.59:0/16876983 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb660066b00 0x7fb66010d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-23T08:50:41.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.714+0000 7fb65effd700 1 --2- 172.21.15.59:0/16876983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6600fff50 0x7fb66010dd90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb654000c00 tx=0x7fb654009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:41.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.715+0000 7fb647fff700 1 -- 172.21.15.59:0/16876983 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb65400e040 con 0x7fb6600fff50 2024-01-23T08:50:41.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.715+0000 7fb665d71700 1 -- 172.21.15.59:0/16876983 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb66010eeb0 con 0x7fb6600fff50 2024-01-23T08:50:41.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.715+0000 7fb647fff700 1 -- 172.21.15.59:0/16876983 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb654009af0 con 0x7fb6600fff50 2024-01-23T08:50:41.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.715+0000 7fb647fff700 1 -- 172.21.15.59:0/16876983 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb65401c630 con 0x7fb6600fff50 2024-01-23T08:50:41.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.715+0000 7fb665d71700 1 -- 172.21.15.59:0/16876983 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb66010f3f0 con 0x7fb6600fff50 2024-01-23T08:50:41.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.716+0000 7fb665d71700 1 -- 172.21.15.59:0/16876983 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6601076d0 con 0x7fb6600fff50 2024-01-23T08:50:41.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.716+0000 7fb647fff700 1 -- 172.21.15.59:0/16876983 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fb65400bc50 con 0x7fb6600fff50 2024-01-23T08:50:41.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.717+0000 7fb647fff700 1 --2- 172.21.15.59:0/16876983 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fb648071780 0x7fb648073c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:41.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.717+0000 7fb647fff700 1 -- 172.21.15.59:0/16876983 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fb654088020 con 0x7fb6600fff50 2024-01-23T08:50:41.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.717+0000 7fb65f7fe700 1 --2- 172.21.15.59:0/16876983 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fb648071780 0x7fb648073c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:41.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.717+0000 7fb65f7fe700 1 --2- 172.21.15.59:0/16876983 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fb648071780 0x7fb648073c30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb650010520 tx=0x7fb65000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:41.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.721+0000 7fb647fff700 1 -- 172.21.15.59:0/16876983 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb654057fd0 con 0x7fb6600fff50 2024-01-23T08:50:41.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.942+0000 7fb665d71700 1 -- 172.21.15.59:0/16876983 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7fb66004bd60 con 0x7fb6600fff50 2024-01-23T08:50:41.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.942+0000 7fb647fff700 1 -- 172.21.15.59:0/16876983 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7fb654057b60 con 0x7fb6600fff50 2024-01-23T08:50:41.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.944+0000 7fb665d71700 1 -- 172.21.15.59:0/16876983 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fb648071780 msgr2=0x7fb648073c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:41.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.944+0000 7fb665d71700 1 --2- 172.21.15.59:0/16876983 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fb648071780 0x7fb648073c30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb650010520 tx=0x7fb65000b040 comp rx=0 tx=0).stop 2024-01-23T08:50:41.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.944+0000 7fb665d71700 1 -- 172.21.15.59:0/16876983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6600fff50 msgr2=0x7fb66010dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:41.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.944+0000 7fb665d71700 1 --2- 172.21.15.59:0/16876983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6600fff50 0x7fb66010dd90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb654000c00 tx=0x7fb654009e30 comp rx=0 tx=0).stop 2024-01-23T08:50:41.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.944+0000 7fb665d71700 1 -- 172.21.15.59:0/16876983 shutdown_connections 2024-01-23T08:50:41.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.944+0000 7fb665d71700 1 --2- 172.21.15.59:0/16876983 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fb648071780 0x7fb648073c30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:41.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.944+0000 7fb665d71700 1 --2- 172.21.15.59:0/16876983 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb660066b00 0x7fb66010d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:41.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.944+0000 7fb665d71700 1 --2- 172.21.15.59:0/16876983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6600fff50 0x7fb66010dd90 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:41.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.944+0000 7fb665d71700 1 -- 172.21.15.59:0/16876983 >> 172.21.15.59:0/16876983 conn(0x7fb6600f7390 msgr2=0x7fb6600f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:41.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.945+0000 7fb665d71700 1 -- 172.21.15.59:0/16876983 shutdown_connections 2024-01-23T08:50:41.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:41.945+0000 7fb665d71700 1 -- 172.21.15.59:0/16876983 wait complete. 2024-01-23T08:50:42.590 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-01-23T08:50:42.894 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:50:43.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:43 smithi059 ceph-mon[28424]: pgmap v201: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-01-23T08:50:43.694 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:43 smithi137 ceph-mon[35787]: pgmap v201: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-01-23T08:50:44.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.680+0000 7f5bdd9a7700 1 -- 172.21.15.59:0/2602929444 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bd80fcee0 msgr2=0x7f5bd80fd330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:44.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.680+0000 7f5bdd9a7700 1 --2- 172.21.15.59:0/2602929444 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bd80fcee0 0x7f5bd80fd330 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc004940 tx=0x7f5bcc009e30 comp rx=0 tx=0).stop 2024-01-23T08:50:44.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.680+0000 7f5bdd9a7700 1 -- 172.21.15.59:0/2602929444 shutdown_connections 2024-01-23T08:50:44.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.680+0000 7f5bdd9a7700 1 --2- 172.21.15.59:0/2602929444 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bd80fcee0 0x7f5bd80fd330 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:44.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.680+0000 7f5bdd9a7700 1 --2- 172.21.15.59:0/2602929444 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bd80fbcd0 0x7f5bd80fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:44.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.680+0000 7f5bdd9a7700 1 -- 172.21.15.59:0/2602929444 >> 172.21.15.59:0/2602929444 conn(0x7f5bd80f73b0 msgr2=0x7f5bd80f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:50:44.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.680+0000 7f5bdd9a7700 1 -- 172.21.15.59:0/2602929444 shutdown_connections 2024-01-23T08:50:44.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.680+0000 7f5bdd9a7700 1 -- 172.21.15.59:0/2602929444 wait complete. 2024-01-23T08:50:44.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.681+0000 7f5bdd9a7700 1 Processor -- start 2024-01-23T08:50:44.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.681+0000 7f5bdd9a7700 1 -- start start 2024-01-23T08:50:44.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.681+0000 7f5bdd9a7700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bd80fbcd0 0x7f5bd810c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:44.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.681+0000 7f5bdd9a7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bd80fcee0 0x7f5bd810cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:44.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.681+0000 7f5bdd9a7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd810d1d0 con 0x7f5bd80fcee0 2024-01-23T08:50:44.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.681+0000 7f5bdd9a7700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd810da70 con 0x7f5bd80fbcd0 2024-01-23T08:50:44.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.682+0000 7f5bd67fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bd80fcee0 0x7f5bd810cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:44.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.682+0000 7f5bd67fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bd80fcee0 0x7f5bd810cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41194/0 (socket says 172.21.15.59:41194) 2024-01-23T08:50:44.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.682+0000 7f5bd67fc700 1 -- 172.21.15.59:0/1922688162 learned_addr learned my addr 172.21.15.59:0/1922688162 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:50:44.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.682+0000 7f5bd6ffd700 1 --2- 172.21.15.59:0/1922688162 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bd80fbcd0 0x7f5bd810c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:44.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.682+0000 7f5bd67fc700 1 -- 172.21.15.59:0/1922688162 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bd80fbcd0 msgr2=0x7f5bd810c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:50:44.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.682+0000 7f5bd67fc700 1 --2- 172.21.15.59:0/1922688162 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bd80fbcd0 0x7f5bd810c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:50:44.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.682+0000 7f5bd67fc700 1 -- 172.21.15.59:0/1922688162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bcc005040 con 0x7f5bd80fcee0 2024-01-23T08:50:44.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.682+0000 7f5bd67fc700 1 --2- 172.21.15.59:0/1922688162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bd80fcee0 0x7f5bd810cbd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc009360 tx=0x7f5bcc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:44.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.682+0000 7f5bbf7fe700 1 -- 172.21.15.59:0/1922688162 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5bcc00e040 con 0x7f5bd80fcee0 2024-01-23T08:50:44.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.682+0000 7f5bdd9a7700 1 -- 172.21.15.59:0/1922688162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd810dcf0 con 0x7f5bd80fcee0 2024-01-23T08:50:44.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.682+0000 7f5bbf7fe700 1 -- 172.21.15.59:0/1922688162 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5bcc00bc50 con 0x7f5bd80fcee0 2024-01-23T08:50:44.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.682+0000 7f5bbf7fe700 1 -- 172.21.15.59:0/1922688162 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5bcc01c3f0 con 0x7f5bd80fcee0 2024-01-23T08:50:44.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.682+0000 7f5bdd9a7700 1 -- 172.21.15.59:0/1922688162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd810e230 con 0x7f5bd80fcee0 2024-01-23T08:50:44.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.683+0000 7f5bdd9a7700 1 -- 172.21.15.59:0/1922688162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bd804bd60 con 0x7f5bd80fcee0 2024-01-23T08:50:44.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.684+0000 7f5bbf7fe700 1 -- 172.21.15.59:0/1922688162 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f5bcc01c6b0 con 0x7f5bd80fcee0 2024-01-23T08:50:44.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.684+0000 7f5bbf7fe700 1 --2- 172.21.15.59:0/1922688162 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f5bc0068f60 0x7f5bc006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:50:44.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.685+0000 7f5bbf7fe700 1 -- 172.21.15.59:0/1922688162 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f5bcc0881a0 con 0x7f5bd80fcee0 2024-01-23T08:50:44.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.685+0000 7f5bd6ffd700 1 --2- 172.21.15.59:0/1922688162 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f5bc0068f60 0x7f5bc006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:50:44.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.685+0000 7f5bd6ffd700 1 --2- 172.21.15.59:0/1922688162 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f5bc0068f60 0x7f5bc006b410 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5bc80041d0 tx=0x7f5bc80043c0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:50:44.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.689+0000 7f5bbf7fe700 1 -- 172.21.15.59:0/1922688162 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5bcc058180 con 0x7f5bd80fcee0 2024-01-23T08:50:44.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:50:44.908+0000 7f5bdd9a7700 1 -- 172.21.15.59:0/1922688162 --> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f5bd810e5d0 con 0x7f5bc0068f60 2024-01-23T08:50:45.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:45 smithi059 ceph-mon[28424]: pgmap v202: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 853 B/s wr, 4 op/s 2024-01-23T08:50:45.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:45 smithi137 ceph-mon[35787]: pgmap v202: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 853 B/s wr, 4 op/s 2024-01-23T08:50:46.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:46 smithi059 ceph-mon[28424]: from='client.14548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:50:46.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:46 smithi137 ceph-mon[35787]: from='client.14548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:50:47.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:47 smithi059 ceph-mon[28424]: pgmap v203: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 5 op/s 2024-01-23T08:50:47.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:47 smithi137 ceph-mon[35787]: pgmap v203: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 5 op/s 2024-01-23T08:50:49.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:49 smithi059 ceph-mon[28424]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2024-01-23T08:50:49.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:49 smithi137 ceph-mon[35787]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2024-01-23T08:50:51.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:51 smithi059 ceph-mon[28424]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-01-23T08:50:51.542 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:51 smithi137 ceph-mon[35787]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-01-23T08:50:53.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:53 smithi059 ceph-mon[28424]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2024-01-23T08:50:53.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:53 smithi137 ceph-mon[35787]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2024-01-23T08:50:55.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:55 smithi059 ceph-mon[28424]: pgmap v207: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-01-23T08:50:55.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:55 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:50:55.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:55 smithi137 ceph-mon[35787]: pgmap v207: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-01-23T08:50:55.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:55 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:50:57.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:57 smithi059 ceph-mon[28424]: pgmap v208: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-01-23T08:50:57.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:57 smithi137 ceph-mon[35787]: pgmap v208: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-01-23T08:50:59.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:50:59 smithi059 ceph-mon[28424]: pgmap v209: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-01-23T08:50:59.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:50:59 smithi137 ceph-mon[35787]: pgmap v209: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-01-23T08:51:01.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:01 smithi059 ceph-mon[28424]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-01-23T08:51:01.542 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:01 smithi137 ceph-mon[35787]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-01-23T08:51:03.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:03 smithi059 ceph-mon[28424]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-01-23T08:51:03.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:03 smithi137 ceph-mon[35787]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-01-23T08:51:05.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:05 smithi059 ceph-mon[28424]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-01-23T08:51:05.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:05 smithi137 ceph-mon[35787]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-01-23T08:51:07.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:07 smithi059 ceph-mon[28424]: pgmap v213: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-01-23T08:51:07.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:07 smithi137 ceph-mon[35787]: pgmap v213: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-01-23T08:51:09.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:09 smithi059 ceph-mon[28424]: pgmap v214: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-01-23T08:51:09.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:09 smithi137 ceph-mon[35787]: pgmap v214: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-01-23T08:51:11.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:11 smithi059 ceph-mon[28424]: pgmap v215: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-01-23T08:51:11.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:11 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:51:11.542 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:11 smithi137 ceph-mon[35787]: pgmap v215: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-01-23T08:51:11.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:11 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:51:13.599 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:13 smithi059 ceph-mon[28424]: pgmap v216: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-01-23T08:51:13.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:13 smithi137 ceph-mon[35787]: pgmap v216: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-01-23T08:51:14.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:14.623+0000 7f5bbf7fe700 1 -- 172.21.15.59:0/1922688162 <== mgr.14210 v2:172.21.15.59:6800/3219735162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f5bd810e5d0 con 0x7f5bc0068f60 2024-01-23T08:51:14.621 INFO:teuthology.orchestra.run.smithi059.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-23T08:51:14.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:14.626+0000 7f5bdd9a7700 1 -- 172.21.15.59:0/1922688162 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f5bc0068f60 msgr2=0x7f5bc006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:14.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:14.626+0000 7f5bdd9a7700 1 --2- 172.21.15.59:0/1922688162 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f5bc0068f60 0x7f5bc006b410 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5bc80041d0 tx=0x7f5bc80043c0 comp rx=0 tx=0).stop 2024-01-23T08:51:14.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:14.626+0000 7f5bdd9a7700 1 -- 172.21.15.59:0/1922688162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bd80fcee0 msgr2=0x7f5bd810cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:14.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:14.626+0000 7f5bdd9a7700 1 --2- 172.21.15.59:0/1922688162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bd80fcee0 0x7f5bd810cbd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc009360 tx=0x7f5bcc009e30 comp rx=0 tx=0).stop 2024-01-23T08:51:14.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:14.626+0000 7f5bdd9a7700 1 -- 172.21.15.59:0/1922688162 shutdown_connections 2024-01-23T08:51:14.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:14.626+0000 7f5bdd9a7700 1 --2- 172.21.15.59:0/1922688162 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f5bc0068f60 0x7f5bc006b410 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:14.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:14.626+0000 7f5bdd9a7700 1 --2- 172.21.15.59:0/1922688162 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bd80fbcd0 0x7f5bd810c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:14.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:14.626+0000 7f5bdd9a7700 1 --2- 172.21.15.59:0/1922688162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bd80fcee0 0x7f5bd810cbd0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:14.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:14.626+0000 7f5bdd9a7700 1 -- 172.21.15.59:0/1922688162 >> 172.21.15.59:0/1922688162 conn(0x7f5bd80f73b0 msgr2=0x7f5bd8100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:51:14.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:14.626+0000 7f5bdd9a7700 1 -- 172.21.15.59:0/1922688162 shutdown_connections 2024-01-23T08:51:14.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:14.627+0000 7f5bdd9a7700 1 -- 172.21.15.59:0/1922688162 wait complete. 2024-01-23T08:51:15.402 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-01-23T08:51:15.621 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:15 smithi059 ceph-mon[28424]: pgmap v217: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-01-23T08:51:15.621 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:15 smithi059 ceph-mon[28424]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-23T08:51:15.621 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:15 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:51:15.621 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:15 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:51:15.622 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:15 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:51:15.622 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:15 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:51:15.622 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:15 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:51:15.762 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:51:16.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:15 smithi137 ceph-mon[35787]: pgmap v217: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-01-23T08:51:16.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:15 smithi137 ceph-mon[35787]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-23T08:51:16.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:15 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:51:16.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:15 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:51:16.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:15 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:51:16.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:15 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-23T08:51:16.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:15 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:51:16.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:16 smithi137 ceph-mon[35787]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-23T08:51:16.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:16 smithi059 ceph-mon[28424]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-23T08:51:17.820 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:17 smithi059 ceph-mon[28424]: pgmap v218: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 5 op/s 2024-01-23T08:51:18.042 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:17 smithi137 ceph-mon[35787]: pgmap v218: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 5 op/s 2024-01-23T08:51:18.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.248+0000 7fa030b9f700 1 -- 172.21.15.59:0/402304456 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa02c0fde00 msgr2=0x7fa02c0fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:18.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.248+0000 7fa030b9f700 1 --2- 172.21.15.59:0/402304456 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa02c0fde00 0x7fa02c0fe250 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa020004940 tx=0x7fa020009e30 comp rx=0 tx=0).stop 2024-01-23T08:51:18.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.248+0000 7fa030b9f700 1 -- 172.21.15.59:0/402304456 shutdown_connections 2024-01-23T08:51:18.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.248+0000 7fa030b9f700 1 --2- 172.21.15.59:0/402304456 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa02c0fde00 0x7fa02c0fe250 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:18.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.248+0000 7fa030b9f700 1 --2- 172.21.15.59:0/402304456 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa02c0fcbf0 0x7fa02c0fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:18.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.248+0000 7fa030b9f700 1 -- 172.21.15.59:0/402304456 >> 172.21.15.59:0/402304456 conn(0x7fa02c0f8290 msgr2=0x7fa02c0fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:51:18.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.248+0000 7fa030b9f700 1 -- 172.21.15.59:0/402304456 shutdown_connections 2024-01-23T08:51:18.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.248+0000 7fa030b9f700 1 -- 172.21.15.59:0/402304456 wait complete. 2024-01-23T08:51:18.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.249+0000 7fa030b9f700 1 Processor -- start 2024-01-23T08:51:18.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.249+0000 7fa030b9f700 1 -- start start 2024-01-23T08:51:18.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.249+0000 7fa030b9f700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa02c0fcbf0 0x7fa02c10d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:18.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.249+0000 7fa030b9f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa02c0fde00 0x7fa02c10d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:18.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.250+0000 7fa02b7fe700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa02c0fcbf0 0x7fa02c10d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:18.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.250+0000 7fa02b7fe700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa02c0fcbf0 0x7fa02c10d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.59:45884/0 (socket says 172.21.15.59:45884) 2024-01-23T08:51:18.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.250+0000 7fa02affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa02c0fde00 0x7fa02c10d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:18.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.250+0000 7fa02affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa02c0fde00 0x7fa02c10d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48614/0 (socket says 172.21.15.59:48614) 2024-01-23T08:51:18.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.250+0000 7fa02b7fe700 1 -- 172.21.15.59:0/635504130 learned_addr learned my addr 172.21.15.59:0/635504130 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:51:18.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.250+0000 7fa030b9f700 1 -- 172.21.15.59:0/635504130 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02c10df80 con 0x7fa02c0fde00 2024-01-23T08:51:18.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.250+0000 7fa030b9f700 1 -- 172.21.15.59:0/635504130 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02c10e820 con 0x7fa02c0fcbf0 2024-01-23T08:51:18.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.250+0000 7fa02b7fe700 1 -- 172.21.15.59:0/635504130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa02c0fde00 msgr2=0x7fa02c10d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:18.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.250+0000 7fa02b7fe700 1 --2- 172.21.15.59:0/635504130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa02c0fde00 0x7fa02c10d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:18.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.250+0000 7fa02b7fe700 1 -- 172.21.15.59:0/635504130 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa020005040 con 0x7fa02c0fcbf0 2024-01-23T08:51:18.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.250+0000 7fa02affd700 1 --2- 172.21.15.59:0/635504130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa02c0fde00 0x7fa02c10d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-23T08:51:18.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.251+0000 7fa02b7fe700 1 --2- 172.21.15.59:0/635504130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa02c0fcbf0 0x7fa02c10d440 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa01c00eb70 tx=0x7fa01c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:51:18.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.251+0000 7fa013fff700 1 -- 172.21.15.59:0/635504130 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa01c007cc0 con 0x7fa02c0fcbf0 2024-01-23T08:51:18.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.251+0000 7fa013fff700 1 -- 172.21.15.59:0/635504130 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa01c00f040 con 0x7fa02c0fcbf0 2024-01-23T08:51:18.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.251+0000 7fa013fff700 1 -- 172.21.15.59:0/635504130 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa01c00c9d0 con 0x7fa02c0fcbf0 2024-01-23T08:51:18.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.251+0000 7fa030b9f700 1 -- 172.21.15.59:0/635504130 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa02c10eaa0 con 0x7fa02c0fcbf0 2024-01-23T08:51:18.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.251+0000 7fa030b9f700 1 -- 172.21.15.59:0/635504130 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa02c10ef90 con 0x7fa02c0fcbf0 2024-01-23T08:51:18.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.252+0000 7fa030b9f700 1 -- 172.21.15.59:0/635504130 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa02c04bd60 con 0x7fa02c0fcbf0 2024-01-23T08:51:18.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.253+0000 7fa013fff700 1 -- 172.21.15.59:0/635504130 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fa01c00cc40 con 0x7fa02c0fcbf0 2024-01-23T08:51:18.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.253+0000 7fa013fff700 1 --2- 172.21.15.59:0/635504130 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fa014068f70 0x7fa01406b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:18.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.254+0000 7fa013fff700 1 -- 172.21.15.59:0/635504130 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa01c0868a0 con 0x7fa02c0fcbf0 2024-01-23T08:51:18.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.254+0000 7fa02affd700 1 --2- 172.21.15.59:0/635504130 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fa014068f70 0x7fa01406b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:18.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.254+0000 7fa02affd700 1 --2- 172.21.15.59:0/635504130 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fa014068f70 0x7fa01406b420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa020013680 tx=0x7fa020009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:51:18.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.258+0000 7fa013fff700 1 -- 172.21.15.59:0/635504130 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa01c04da00 con 0x7fa02c0fcbf0 2024-01-23T08:51:18.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.682+0000 7fa030b9f700 1 -- 172.21.15.59:0/635504130 --> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa02c102750 con 0x7fa014068f70 2024-01-23T08:51:18.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.683+0000 7fa013fff700 1 -- 172.21.15.59:0/635504130 <== mgr.14210 v2:172.21.15.59:6800/3219735162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7fa02c102750 con 0x7fa014068f70 2024-01-23T08:51:18.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.686+0000 7fa030b9f700 1 -- 172.21.15.59:0/635504130 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fa014068f70 msgr2=0x7fa01406b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:18.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.686+0000 7fa030b9f700 1 --2- 172.21.15.59:0/635504130 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fa014068f70 0x7fa01406b420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa020013680 tx=0x7fa020009e30 comp rx=0 tx=0).stop 2024-01-23T08:51:18.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.686+0000 7fa030b9f700 1 -- 172.21.15.59:0/635504130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa02c0fcbf0 msgr2=0x7fa02c10d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:18.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.686+0000 7fa030b9f700 1 --2- 172.21.15.59:0/635504130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa02c0fcbf0 0x7fa02c10d440 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa01c00eb70 tx=0x7fa01c00b040 comp rx=0 tx=0).stop 2024-01-23T08:51:18.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.686+0000 7fa030b9f700 1 -- 172.21.15.59:0/635504130 shutdown_connections 2024-01-23T08:51:18.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.686+0000 7fa030b9f700 1 --2- 172.21.15.59:0/635504130 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fa014068f70 0x7fa01406b420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:18.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.686+0000 7fa030b9f700 1 --2- 172.21.15.59:0/635504130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa02c0fcbf0 0x7fa02c10d440 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:18.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.686+0000 7fa030b9f700 1 --2- 172.21.15.59:0/635504130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa02c0fde00 0x7fa02c10d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:18.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.686+0000 7fa030b9f700 1 -- 172.21.15.59:0/635504130 >> 172.21.15.59:0/635504130 conn(0x7fa02c0f8290 msgr2=0x7fa02c101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:51:18.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.686+0000 7fa030b9f700 1 -- 172.21.15.59:0/635504130 shutdown_connections 2024-01-23T08:51:18.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.686+0000 7fa030b9f700 1 -- 172.21.15.59:0/635504130 wait complete. 2024-01-23T08:51:18.698 INFO:teuthology.orchestra.run.smithi059.stdout:true 2024-01-23T08:51:18.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.850+0000 7ff9a62b0700 1 -- 172.21.15.59:0/4136336264 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9a00fff50 msgr2=0x7ff9a01023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:18.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.850+0000 7ff9a62b0700 1 --2- 172.21.15.59:0/4136336264 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9a00fff50 0x7ff9a01023b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff994004940 tx=0x7ff994009e30 comp rx=0 tx=0).stop 2024-01-23T08:51:18.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.851+0000 7ff9a62b0700 1 -- 172.21.15.59:0/4136336264 shutdown_connections 2024-01-23T08:51:18.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.851+0000 7ff9a62b0700 1 --2- 172.21.15.59:0/4136336264 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9a00fff50 0x7ff9a01023b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:18.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.851+0000 7ff9a62b0700 1 --2- 172.21.15.59:0/4136336264 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff9a0066270 0x7ff9a00ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:18.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.851+0000 7ff9a62b0700 1 -- 172.21.15.59:0/4136336264 >> 172.21.15.59:0/4136336264 conn(0x7ff9a00f7370 msgr2=0x7ff9a00f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:51:18.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.851+0000 7ff9a62b0700 1 -- 172.21.15.59:0/4136336264 shutdown_connections 2024-01-23T08:51:18.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.851+0000 7ff9a62b0700 1 -- 172.21.15.59:0/4136336264 wait complete. 2024-01-23T08:51:18.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.852+0000 7ff9a62b0700 1 Processor -- start 2024-01-23T08:51:18.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.853+0000 7ff9a62b0700 1 -- start start 2024-01-23T08:51:18.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.853+0000 7ff9a62b0700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff9a0066270 0x7ff9a010d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:18.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.853+0000 7ff9a62b0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9a00fff50 0x7ff9a010db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:18.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.853+0000 7ff99ffff700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff9a0066270 0x7ff9a010d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:18.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.853+0000 7ff99ffff700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff9a0066270 0x7ff9a010d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.59:45892/0 (socket says 172.21.15.59:45892) 2024-01-23T08:51:18.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.854+0000 7ff99ffff700 1 -- 172.21.15.59:0/3053458567 learned_addr learned my addr 172.21.15.59:0/3053458567 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:51:18.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.854+0000 7ff9a62b0700 1 -- 172.21.15.59:0/3053458567 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a010e140 con 0x7ff9a00fff50 2024-01-23T08:51:18.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.854+0000 7ff9a62b0700 1 -- 172.21.15.59:0/3053458567 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a010e9e0 con 0x7ff9a0066270 2024-01-23T08:51:18.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.854+0000 7ff99f7fe700 1 --2- 172.21.15.59:0/3053458567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9a00fff50 0x7ff9a010db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:18.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.854+0000 7ff99ffff700 1 -- 172.21.15.59:0/3053458567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9a00fff50 msgr2=0x7ff9a010db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:18.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.854+0000 7ff99ffff700 1 --2- 172.21.15.59:0/3053458567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9a00fff50 0x7ff9a010db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:18.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.854+0000 7ff99ffff700 1 -- 172.21.15.59:0/3053458567 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff994005040 con 0x7ff9a0066270 2024-01-23T08:51:18.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.854+0000 7ff99f7fe700 1 --2- 172.21.15.59:0/3053458567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9a00fff50 0x7ff9a010db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-23T08:51:18.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.855+0000 7ff99ffff700 1 --2- 172.21.15.59:0/3053458567 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff9a0066270 0x7ff9a010d600 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff99000eb40 tx=0x7ff99000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:51:18.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.855+0000 7ff99cff9700 1 -- 172.21.15.59:0/3053458567 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff990007cc0 con 0x7ff9a0066270 2024-01-23T08:51:18.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.855+0000 7ff9a62b0700 1 -- 172.21.15.59:0/3053458567 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a010ec60 con 0x7ff9a0066270 2024-01-23T08:51:18.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.855+0000 7ff9a62b0700 1 -- 172.21.15.59:0/3053458567 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a010f1a0 con 0x7ff9a0066270 2024-01-23T08:51:18.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.855+0000 7ff99cff9700 1 -- 172.21.15.59:0/3053458567 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff99000f040 con 0x7ff9a0066270 2024-01-23T08:51:18.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.855+0000 7ff99cff9700 1 -- 172.21.15.59:0/3053458567 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff990009570 con 0x7ff9a0066270 2024-01-23T08:51:18.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.856+0000 7ff9867fc700 1 -- 172.21.15.59:0/3053458567 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9a004bd60 con 0x7ff9a0066270 2024-01-23T08:51:18.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.857+0000 7ff99cff9700 1 -- 172.21.15.59:0/3053458567 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7ff9900096d0 con 0x7ff9a0066270 2024-01-23T08:51:18.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.858+0000 7ff99cff9700 1 --2- 172.21.15.59:0/3053458567 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7ff988068f70 0x7ff98806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:18.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.858+0000 7ff99cff9700 1 -- 172.21.15.59:0/3053458567 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff990086a70 con 0x7ff9a0066270 2024-01-23T08:51:18.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.858+0000 7ff99f7fe700 1 --2- 172.21.15.59:0/3053458567 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7ff988068f70 0x7ff98806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:18.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.859+0000 7ff99f7fe700 1 --2- 172.21.15.59:0/3053458567 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7ff988068f70 0x7ff98806b420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff994000c00 tx=0x7ff994009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:51:18.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:18.863+0000 7ff99cff9700 1 -- 172.21.15.59:0/3053458567 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ff99004da00 con 0x7ff9a0066270 2024-01-23T08:51:19.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.281+0000 7ff9867fc700 1 -- 172.21.15.59:0/3053458567 --> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9a0105760 con 0x7ff988068f70 2024-01-23T08:51:19.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.283+0000 7ff99cff9700 1 -- 172.21.15.59:0/3053458567 <== mgr.14210 v2:172.21.15.59:6800/3219735162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7ff9a0105760 con 0x7ff988068f70 2024-01-23T08:51:19.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.286+0000 7ff9867fc700 1 -- 172.21.15.59:0/3053458567 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7ff988068f70 msgr2=0x7ff98806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:19.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.286+0000 7ff9867fc700 1 --2- 172.21.15.59:0/3053458567 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7ff988068f70 0x7ff98806b420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff994000c00 tx=0x7ff994009e30 comp rx=0 tx=0).stop 2024-01-23T08:51:19.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.286+0000 7ff9867fc700 1 -- 172.21.15.59:0/3053458567 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff9a0066270 msgr2=0x7ff9a010d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:19.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.286+0000 7ff9867fc700 1 --2- 172.21.15.59:0/3053458567 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff9a0066270 0x7ff9a010d600 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff99000eb40 tx=0x7ff99000b040 comp rx=0 tx=0).stop 2024-01-23T08:51:19.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.286+0000 7ff9867fc700 1 -- 172.21.15.59:0/3053458567 shutdown_connections 2024-01-23T08:51:19.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.286+0000 7ff9867fc700 1 --2- 172.21.15.59:0/3053458567 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7ff988068f70 0x7ff98806b420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:19.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.286+0000 7ff9867fc700 1 --2- 172.21.15.59:0/3053458567 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff9a0066270 0x7ff9a010d600 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:19.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.286+0000 7ff9867fc700 1 --2- 172.21.15.59:0/3053458567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9a00fff50 0x7ff9a010db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:19.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.286+0000 7ff9867fc700 1 -- 172.21.15.59:0/3053458567 >> 172.21.15.59:0/3053458567 conn(0x7ff9a00f7370 msgr2=0x7ff9a00f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:51:19.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.286+0000 7ff9867fc700 1 -- 172.21.15.59:0/3053458567 shutdown_connections 2024-01-23T08:51:19.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.286+0000 7ff9867fc700 1 -- 172.21.15.59:0/3053458567 wait complete. 2024-01-23T08:51:19.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.453+0000 7fbbf7d3e700 1 -- 172.21.15.59:0/1415189049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbf00fff00 msgr2=0x7fbbf0102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:19.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.453+0000 7fbbf7d3e700 1 --2- 172.21.15.59:0/1415189049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbf00fff00 0x7fbbf0102340 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0004940 tx=0x7fbbe0009e30 comp rx=0 tx=0).stop 2024-01-23T08:51:19.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.454+0000 7fbbf7d3e700 1 -- 172.21.15.59:0/1415189049 shutdown_connections 2024-01-23T08:51:19.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.454+0000 7fbbf7d3e700 1 --2- 172.21.15.59:0/1415189049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbf00fff00 0x7fbbf0102340 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:19.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.454+0000 7fbbf7d3e700 1 --2- 172.21.15.59:0/1415189049 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbbf0066ae0 0x7fbbf00ff930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:19.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.454+0000 7fbbf7d3e700 1 -- 172.21.15.59:0/1415189049 >> 172.21.15.59:0/1415189049 conn(0x7fbbf00f7320 msgr2=0x7fbbf00f9750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:51:19.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.454+0000 7fbbf7d3e700 1 -- 172.21.15.59:0/1415189049 shutdown_connections 2024-01-23T08:51:19.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.454+0000 7fbbf7d3e700 1 -- 172.21.15.59:0/1415189049 wait complete. 2024-01-23T08:51:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.455+0000 7fbbf7d3e700 1 Processor -- start 2024-01-23T08:51:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.455+0000 7fbbf7d3e700 1 -- start start 2024-01-23T08:51:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.455+0000 7fbbf7d3e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbf0066ae0 0x7fbbf00fd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.455+0000 7fbbf7d3e700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbbf00fff00 0x7fbbf00fb830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.455+0000 7fbbf7d3e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf0111060 con 0x7fbbf0066ae0 2024-01-23T08:51:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.455+0000 7fbbf7d3e700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf01111d0 con 0x7fbbf00fff00 2024-01-23T08:51:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.455+0000 7fbbf5ada700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbf0066ae0 0x7fbbf00fd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.455+0000 7fbbf52d9700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbbf00fff00 0x7fbbf00fb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.455+0000 7fbbf5ada700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbf0066ae0 0x7fbbf00fd3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48662/0 (socket says 172.21.15.59:48662) 2024-01-23T08:51:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.455+0000 7fbbf5ada700 1 -- 172.21.15.59:0/53406041 learned_addr learned my addr 172.21.15.59:0/53406041 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:51:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.456+0000 7fbbf5ada700 1 -- 172.21.15.59:0/53406041 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbbf00fff00 msgr2=0x7fbbf00fb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.456+0000 7fbbf5ada700 1 --2- 172.21.15.59:0/53406041 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbbf00fff00 0x7fbbf00fb830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.456+0000 7fbbf5ada700 1 -- 172.21.15.59:0/53406041 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe0005040 con 0x7fbbf0066ae0 2024-01-23T08:51:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.456+0000 7fbbf52d9700 1 --2- 172.21.15.59:0/53406041 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbbf00fff00 0x7fbbf00fb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-23T08:51:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.456+0000 7fbbf5ada700 1 --2- 172.21.15.59:0/53406041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbf0066ae0 0x7fbbf00fd3b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbbec00eb40 tx=0x7fbbec00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:51:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.456+0000 7fbbe67fc700 1 -- 172.21.15.59:0/53406041 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbec007cc0 con 0x7fbbf0066ae0 2024-01-23T08:51:19.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.456+0000 7fbbe67fc700 1 -- 172.21.15.59:0/53406041 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbbec00f040 con 0x7fbbf0066ae0 2024-01-23T08:51:19.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.456+0000 7fbbe67fc700 1 -- 172.21.15.59:0/53406041 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbec00c990 con 0x7fbbf0066ae0 2024-01-23T08:51:19.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.456+0000 7fbbf7d3e700 1 -- 172.21.15.59:0/53406041 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf00fbe30 con 0x7fbbf0066ae0 2024-01-23T08:51:19.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.456+0000 7fbbf7d3e700 1 -- 172.21.15.59:0/53406041 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf00fc370 con 0x7fbbf0066ae0 2024-01-23T08:51:19.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.458+0000 7fbbf7d3e700 1 -- 172.21.15.59:0/53406041 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf0107730 con 0x7fbbf0066ae0 2024-01-23T08:51:19.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.459+0000 7fbbe67fc700 1 -- 172.21.15.59:0/53406041 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fbbec00caf0 con 0x7fbbf0066ae0 2024-01-23T08:51:19.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.460+0000 7fbbe67fc700 1 --2- 172.21.15.59:0/53406041 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbbdc069020 0x7fbbdc06b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:19.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.460+0000 7fbbe67fc700 1 -- 172.21.15.59:0/53406041 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fbbec086b10 con 0x7fbbf0066ae0 2024-01-23T08:51:19.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.460+0000 7fbbf52d9700 1 --2- 172.21.15.59:0/53406041 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbbdc069020 0x7fbbdc06b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:19.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.461+0000 7fbbf52d9700 1 --2- 172.21.15.59:0/53406041 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbbdc069020 0x7fbbdc06b4d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0013680 tx=0x7fbbe0009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:51:19.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.467+0000 7fbbe67fc700 1 -- 172.21.15.59:0/53406041 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fbbec056ac0 con 0x7fbbf0066ae0 2024-01-23T08:51:19.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.871+0000 7fbbf7d3e700 1 -- 172.21.15.59:0/53406041 --> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbf0105880 con 0x7fbbdc069020 2024-01-23T08:51:19.886 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:19 smithi059 ceph-mon[28424]: pgmap v219: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2024-01-23T08:51:19.886 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:19 smithi059 ceph-mon[28424]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:51:19.887 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:19 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:51:19.887 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:19 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:51:19.887 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:19 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:51:19.887 INFO:teuthology.orchestra.run.smithi059.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-01-23T08:51:19.887 INFO:teuthology.orchestra.run.smithi059.stdout:alertmanager.smithi059 smithi059 *:9093,9094 running (5m) 54s ago 6m 25.7M - 0.25.0 c8568f914cd2 1cc540895329 2024-01-23T08:51:19.887 INFO:teuthology.orchestra.run.smithi059.stdout:crash.smithi059 smithi059 running (6m) 54s ago 6m 6656k - 17.2.7-397-gb824e5be 61b360e6ff65 12d41238ee1c 2024-01-23T08:51:19.887 INFO:teuthology.orchestra.run.smithi059.stdout:crash.smithi137 smithi137 running (5m) 59s ago 5m 6665k - 17.2.7-397-gb824e5be 61b360e6ff65 62f889b4ea0f 2024-01-23T08:51:19.887 INFO:teuthology.orchestra.run.smithi059.stdout:grafana.smithi059 smithi059 *:3000 running (5m) 54s ago 6m 88.1M - 9.4.7 954c08fa6188 f2e5dfb17ced 2024-01-23T08:51:19.887 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi059.lsxsmm smithi059 running (79s) 54s ago 78s 11.5M - 17.2.7-397-gb824e5be 61b360e6ff65 f35175705b99 2024-01-23T08:51:19.887 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi059.twpxfz smithi059 running (72s) 54s ago 71s 14.8M - 17.2.7-397-gb824e5be 61b360e6ff65 bbe0beb326d2 2024-01-23T08:51:19.887 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi137.mgacmp smithi137 running (76s) 59s ago 76s 14.5M - 17.2.7-397-gb824e5be 61b360e6ff65 57bc18125785 2024-01-23T08:51:19.887 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi137.scidwx smithi137 running (69s) 59s ago 69s 15.5M - 17.2.7-397-gb824e5be 61b360e6ff65 79e1d1658276 2024-01-23T08:51:19.888 INFO:teuthology.orchestra.run.smithi059.stdout:mgr.smithi059.bzqirf smithi059 *:9283 running (8m) 54s ago 8m 481M - 17.2.7-397-gb824e5be 61b360e6ff65 88cf77d2510a 2024-01-23T08:51:19.888 INFO:teuthology.orchestra.run.smithi059.stdout:mgr.smithi137.ooirrh smithi137 *:8443,9283 running (5m) 59s ago 5m 431M - 17.2.7-397-gb824e5be 61b360e6ff65 5af177b46b24 2024-01-23T08:51:19.888 INFO:teuthology.orchestra.run.smithi059.stdout:mon.smithi059 smithi059 running (8m) 54s ago 8m 46.5M 2048M 17.2.7-397-gb824e5be 61b360e6ff65 810a40012e10 2024-01-23T08:51:19.888 INFO:teuthology.orchestra.run.smithi059.stdout:mon.smithi137 smithi137 running (5m) 59s ago 5m 36.3M 2048M 17.2.7-397-gb824e5be 61b360e6ff65 e1d4c945bf49 2024-01-23T08:51:19.888 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter.smithi059 smithi059 *:9100 running (6m) 54s ago 6m 15.9M - 1.5.0 0da6a335fe13 89ec78f972e4 2024-01-23T08:51:19.888 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter.smithi137 smithi137 *:9100 running (5m) 59s ago 5m 15.1M - 1.5.0 0da6a335fe13 1b38a323818a 2024-01-23T08:51:19.888 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 smithi059 running (5m) 54s ago 4m 46.6M 4289M 17.2.7-397-gb824e5be 61b360e6ff65 11074a2f09ab 2024-01-23T08:51:19.888 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 smithi059 running (4m) 54s ago 4m 48.8M 4289M 17.2.7-397-gb824e5be 61b360e6ff65 efe49a487df2 2024-01-23T08:51:19.888 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 smithi059 running (4m) 54s ago 4m 47.4M 4289M 17.2.7-397-gb824e5be 61b360e6ff65 0e2f974c19f0 2024-01-23T08:51:19.888 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 smithi137 running (3m) 59s ago 3m 44.4M 5312M 17.2.7-397-gb824e5be 61b360e6ff65 a7de37e3a235 2024-01-23T08:51:19.888 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 smithi137 running (3m) 59s ago 3m 42.8M 5312M 17.2.7-397-gb824e5be 61b360e6ff65 017510d7e50d 2024-01-23T08:51:19.888 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 smithi137 running (2m) 59s ago 2m 42.5M 5312M 17.2.7-397-gb824e5be 61b360e6ff65 c37971a0dbb9 2024-01-23T08:51:19.888 INFO:teuthology.orchestra.run.smithi059.stdout:prometheus.smithi059 smithi059 *:9095 running (5m) 54s ago 6m 52.1M - 2.43.0 a07b618ecd1d 21db8bea28d9 2024-01-23T08:51:19.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.889+0000 7fbbe67fc700 1 -- 172.21.15.59:0/53406041 <== mgr.14210 v2:172.21.15.59:6800/3219735162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7fbbf0105880 con 0x7fbbdc069020 2024-01-23T08:51:19.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.892+0000 7fbbf7d3e700 1 -- 172.21.15.59:0/53406041 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbbdc069020 msgr2=0x7fbbdc06b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:19.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.892+0000 7fbbf7d3e700 1 --2- 172.21.15.59:0/53406041 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbbdc069020 0x7fbbdc06b4d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0013680 tx=0x7fbbe0009e30 comp rx=0 tx=0).stop 2024-01-23T08:51:19.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.892+0000 7fbbf7d3e700 1 -- 172.21.15.59:0/53406041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbf0066ae0 msgr2=0x7fbbf00fd3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:19.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.892+0000 7fbbf7d3e700 1 --2- 172.21.15.59:0/53406041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbf0066ae0 0x7fbbf00fd3b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbbec00eb40 tx=0x7fbbec00b040 comp rx=0 tx=0).stop 2024-01-23T08:51:19.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.892+0000 7fbbf7d3e700 1 -- 172.21.15.59:0/53406041 shutdown_connections 2024-01-23T08:51:19.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.892+0000 7fbbf7d3e700 1 --2- 172.21.15.59:0/53406041 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbbdc069020 0x7fbbdc06b4d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:19.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.892+0000 7fbbf7d3e700 1 --2- 172.21.15.59:0/53406041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbf0066ae0 0x7fbbf00fd3b0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:19.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.892+0000 7fbbf7d3e700 1 --2- 172.21.15.59:0/53406041 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbbf00fff00 0x7fbbf00fb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:19.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.892+0000 7fbbf7d3e700 1 -- 172.21.15.59:0/53406041 >> 172.21.15.59:0/53406041 conn(0x7fbbf00f7320 msgr2=0x7fbbf00f9750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:51:19.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.892+0000 7fbbf7d3e700 1 -- 172.21.15.59:0/53406041 shutdown_connections 2024-01-23T08:51:19.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:19.892+0000 7fbbf7d3e700 1 -- 172.21.15.59:0/53406041 wait complete. 2024-01-23T08:51:20.042 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:19 smithi137 ceph-mon[35787]: pgmap v219: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2024-01-23T08:51:20.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:19 smithi137 ceph-mon[35787]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:51:20.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:19 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:51:20.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:19 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-23T08:51:20.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:19 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:51:20.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.054+0000 7fc7c5909700 1 -- 172.21.15.59:0/874955090 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7c00fd300 msgr2=0x7fc7c00fd710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:20.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.054+0000 7fc7c5909700 1 --2- 172.21.15.59:0/874955090 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7c00fd300 0x7fc7c00fd710 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0004940 tx=0x7fc7b0009e30 comp rx=0 tx=0).stop 2024-01-23T08:51:20.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.054+0000 7fc7c5909700 1 -- 172.21.15.59:0/874955090 shutdown_connections 2024-01-23T08:51:20.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.054+0000 7fc7c5909700 1 --2- 172.21.15.59:0/874955090 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc7c00fc010 0x7fc7c00fc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:20.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.054+0000 7fc7c5909700 1 --2- 172.21.15.59:0/874955090 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7c00fd300 0x7fc7c00fd710 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:20.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.055+0000 7fc7c5909700 1 -- 172.21.15.59:0/874955090 >> 172.21.15.59:0/874955090 conn(0x7fc7c00f6b30 msgr2=0x7fc7c00f8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:51:20.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.055+0000 7fc7c5909700 1 -- 172.21.15.59:0/874955090 shutdown_connections 2024-01-23T08:51:20.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.055+0000 7fc7c5909700 1 -- 172.21.15.59:0/874955090 wait complete. 2024-01-23T08:51:20.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.056+0000 7fc7c5909700 1 Processor -- start 2024-01-23T08:51:20.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.056+0000 7fc7c5909700 1 -- start start 2024-01-23T08:51:20.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.057+0000 7fc7c5909700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7c00fc010 0x7fc7c010c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:20.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.057+0000 7fc7c5909700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc7c00fd300 0x7fc7c010cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:20.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.057+0000 7fc7c5909700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c010d1b0 con 0x7fc7c00fc010 2024-01-23T08:51:20.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.057+0000 7fc7c5909700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c010da50 con 0x7fc7c00fd300 2024-01-23T08:51:20.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.057+0000 7fc7beffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7c00fc010 0x7fc7c010c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:20.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.057+0000 7fc7beffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7c00fc010 0x7fc7c010c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48684/0 (socket says 172.21.15.59:48684) 2024-01-23T08:51:20.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.057+0000 7fc7beffd700 1 -- 172.21.15.59:0/278273241 learned_addr learned my addr 172.21.15.59:0/278273241 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:51:20.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.057+0000 7fc7beffd700 1 -- 172.21.15.59:0/278273241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc7c00fd300 msgr2=0x7fc7c010cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:20.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.057+0000 7fc7beffd700 1 --2- 172.21.15.59:0/278273241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc7c00fd300 0x7fc7c010cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:20.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.058+0000 7fc7beffd700 1 -- 172.21.15.59:0/278273241 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b0005040 con 0x7fc7c00fc010 2024-01-23T08:51:20.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.058+0000 7fc7beffd700 1 --2- 172.21.15.59:0/278273241 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7c00fc010 0x7fc7c010c670 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0000c00 tx=0x7fc7b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:51:20.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.058+0000 7fc7a77fe700 1 -- 172.21.15.59:0/278273241 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7b0021070 con 0x7fc7c00fc010 2024-01-23T08:51:20.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.058+0000 7fc7c5909700 1 -- 172.21.15.59:0/278273241 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7c010dcd0 con 0x7fc7c00fc010 2024-01-23T08:51:20.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.058+0000 7fc7c5909700 1 -- 172.21.15.59:0/278273241 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7c010e210 con 0x7fc7c00fc010 2024-01-23T08:51:20.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.058+0000 7fc7a77fe700 1 -- 172.21.15.59:0/278273241 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7b001c400 con 0x7fc7c00fc010 2024-01-23T08:51:20.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.058+0000 7fc7a77fe700 1 -- 172.21.15.59:0/278273241 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7b000e070 con 0x7fc7c00fc010 2024-01-23T08:51:20.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.060+0000 7fc7c5909700 1 -- 172.21.15.59:0/278273241 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7ac0052e0 con 0x7fc7c00fc010 2024-01-23T08:51:20.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.061+0000 7fc7a77fe700 1 -- 172.21.15.59:0/278273241 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fc7b0013a30 con 0x7fc7c00fc010 2024-01-23T08:51:20.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.061+0000 7fc7a77fe700 1 --2- 172.21.15.59:0/278273241 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fc7a8068f60 0x7fc7a806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:20.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.062+0000 7fc7a77fe700 1 -- 172.21.15.59:0/278273241 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc7b0058010 con 0x7fc7c00fc010 2024-01-23T08:51:20.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.062+0000 7fc7be7fc700 1 --2- 172.21.15.59:0/278273241 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fc7a8068f60 0x7fc7a806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:20.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.062+0000 7fc7be7fc700 1 --2- 172.21.15.59:0/278273241 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fc7a8068f60 0x7fc7a806b410 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4009fd0 tx=0x7fc7b400b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:51:20.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.070+0000 7fc7a77fe700 1 -- 172.21.15.59:0/278273241 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc7b00576e0 con 0x7fc7c00fc010 2024-01-23T08:51:20.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.623+0000 7fc7c5909700 1 -- 172.21.15.59:0/278273241 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc7ac005860 con 0x7fc7c00fc010 2024-01-23T08:51:20.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.623+0000 7fc7a77fe700 1 -- 172.21.15.59:0/278273241 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7fc7b0057270 con 0x7fc7c00fc010 2024-01-23T08:51:20.621 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-01-23T08:51:20.621 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": { 2024-01-23T08:51:20.621 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-397-gb824e5be (b824e5befce1d427fda2bc90ae52665f0c7e2770) quincy (stable)": 2 2024-01-23T08:51:20.621 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-01-23T08:51:20.621 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": { 2024-01-23T08:51:20.621 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-397-gb824e5be (b824e5befce1d427fda2bc90ae52665f0c7e2770) quincy (stable)": 2 2024-01-23T08:51:20.621 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-01-23T08:51:20.621 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": { 2024-01-23T08:51:20.621 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-397-gb824e5be (b824e5befce1d427fda2bc90ae52665f0c7e2770) quincy (stable)": 6 2024-01-23T08:51:20.622 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-01-23T08:51:20.622 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": { 2024-01-23T08:51:20.622 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-397-gb824e5be (b824e5befce1d427fda2bc90ae52665f0c7e2770) quincy (stable)": 4 2024-01-23T08:51:20.622 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-01-23T08:51:20.622 INFO:teuthology.orchestra.run.smithi059.stdout: "overall": { 2024-01-23T08:51:20.622 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-397-gb824e5be (b824e5befce1d427fda2bc90ae52665f0c7e2770) quincy (stable)": 14 2024-01-23T08:51:20.622 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-01-23T08:51:20.622 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-01-23T08:51:20.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.626+0000 7fc7c5909700 1 -- 172.21.15.59:0/278273241 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fc7a8068f60 msgr2=0x7fc7a806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:20.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.626+0000 7fc7c5909700 1 --2- 172.21.15.59:0/278273241 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fc7a8068f60 0x7fc7a806b410 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4009fd0 tx=0x7fc7b400b040 comp rx=0 tx=0).stop 2024-01-23T08:51:20.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.626+0000 7fc7c5909700 1 -- 172.21.15.59:0/278273241 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7c00fc010 msgr2=0x7fc7c010c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:20.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.626+0000 7fc7c5909700 1 --2- 172.21.15.59:0/278273241 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7c00fc010 0x7fc7c010c670 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0000c00 tx=0x7fc7b0009e30 comp rx=0 tx=0).stop 2024-01-23T08:51:20.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.626+0000 7fc7c5909700 1 -- 172.21.15.59:0/278273241 shutdown_connections 2024-01-23T08:51:20.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.626+0000 7fc7c5909700 1 --2- 172.21.15.59:0/278273241 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fc7a8068f60 0x7fc7a806b410 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:20.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.626+0000 7fc7c5909700 1 --2- 172.21.15.59:0/278273241 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7c00fc010 0x7fc7c010c670 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:20.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.626+0000 7fc7c5909700 1 --2- 172.21.15.59:0/278273241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc7c00fd300 0x7fc7c010cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:20.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.626+0000 7fc7c5909700 1 -- 172.21.15.59:0/278273241 >> 172.21.15.59:0/278273241 conn(0x7fc7c00f6b30 msgr2=0x7fc7c0100250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:51:20.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.626+0000 7fc7c5909700 1 -- 172.21.15.59:0/278273241 shutdown_connections 2024-01-23T08:51:20.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.626+0000 7fc7c5909700 1 -- 172.21.15.59:0/278273241 wait complete. 2024-01-23T08:51:20.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:20 smithi059 ceph-mon[28424]: Upgrade: Target is version 19.0.0-766-g0edf41a6 (unknown) 2024-01-23T08:51:20.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:20 smithi059 ceph-mon[28424]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7, digests ['quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7'] 2024-01-23T08:51:20.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:20 smithi059 ceph-mon[28424]: Upgrade: Need to upgrade myself (mgr.smithi059.bzqirf) 2024-01-23T08:51:20.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:20 smithi059 ceph-mon[28424]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7 on smithi137 2024-01-23T08:51:20.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:20 smithi059 ceph-mon[28424]: from='client.24339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:51:20.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:20 smithi059 ceph-mon[28424]: from='client.? 172.21.15.59:0/278273241' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-01-23T08:51:20.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.733+0000 7f4ddd476700 1 -- 172.21.15.59:0/1292775831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4dd80fbcd0 msgr2=0x7f4dd80fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:20.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.733+0000 7f4ddd476700 1 --2- 172.21.15.59:0/1292775831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4dd80fbcd0 0x7f4dd80fc0e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4dc8004940 tx=0x7f4dc8009e30 comp rx=0 tx=0).stop 2024-01-23T08:51:20.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.733+0000 7f4ddd476700 1 -- 172.21.15.59:0/1292775831 shutdown_connections 2024-01-23T08:51:20.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.733+0000 7f4ddd476700 1 --2- 172.21.15.59:0/1292775831 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4dd80fcee0 0x7f4dd80fd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:20.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.733+0000 7f4ddd476700 1 --2- 172.21.15.59:0/1292775831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4dd80fbcd0 0x7f4dd80fc0e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:20.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.733+0000 7f4ddd476700 1 -- 172.21.15.59:0/1292775831 >> 172.21.15.59:0/1292775831 conn(0x7f4dd80f7390 msgr2=0x7f4dd80f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:51:20.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.733+0000 7f4ddd476700 1 -- 172.21.15.59:0/1292775831 shutdown_connections 2024-01-23T08:51:20.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.733+0000 7f4ddd476700 1 -- 172.21.15.59:0/1292775831 wait complete. 2024-01-23T08:51:20.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.734+0000 7f4ddd476700 1 Processor -- start 2024-01-23T08:51:20.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.734+0000 7f4ddd476700 1 -- start start 2024-01-23T08:51:20.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.734+0000 7f4ddd476700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4dd80fbcd0 0x7f4dd81103f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:20.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.734+0000 7f4ddd476700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4dd80fcee0 0x7f4dd8112940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:20.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.734+0000 7f4ddd476700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd8066b80 con 0x7f4dd80fcee0 2024-01-23T08:51:20.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.734+0000 7f4ddd476700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd8066cc0 con 0x7f4dd80fbcd0 2024-01-23T08:51:20.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.735+0000 7f4dd67fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4dd80fcee0 0x7f4dd8112940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:20.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.735+0000 7f4dd6ffd700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4dd80fbcd0 0x7f4dd81103f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:20.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.735+0000 7f4dd67fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4dd80fcee0 0x7f4dd8112940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48710/0 (socket says 172.21.15.59:48710) 2024-01-23T08:51:20.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.735+0000 7f4dd67fc700 1 -- 172.21.15.59:0/3860093660 learned_addr learned my addr 172.21.15.59:0/3860093660 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:51:20.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.735+0000 7f4dd67fc700 1 -- 172.21.15.59:0/3860093660 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4dd80fbcd0 msgr2=0x7f4dd81103f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:20.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.735+0000 7f4dd67fc700 1 --2- 172.21.15.59:0/3860093660 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4dd80fbcd0 0x7f4dd81103f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:20.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.735+0000 7f4dd67fc700 1 -- 172.21.15.59:0/3860093660 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc8005040 con 0x7f4dd80fcee0 2024-01-23T08:51:20.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.735+0000 7f4dd6ffd700 1 --2- 172.21.15.59:0/3860093660 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4dd80fbcd0 0x7f4dd81103f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-23T08:51:20.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.735+0000 7f4dd67fc700 1 --2- 172.21.15.59:0/3860093660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4dd80fcee0 0x7f4dd8112940 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc00eb70 tx=0x7f4dcc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:51:20.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.735+0000 7f4dbf7fe700 1 -- 172.21.15.59:0/3860093660 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4dcc007cc0 con 0x7f4dd80fcee0 2024-01-23T08:51:20.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.736+0000 7f4dbf7fe700 1 -- 172.21.15.59:0/3860093660 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4dcc00f040 con 0x7f4dd80fcee0 2024-01-23T08:51:20.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.736+0000 7f4ddd476700 1 -- 172.21.15.59:0/3860093660 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd8112f40 con 0x7f4dd80fcee0 2024-01-23T08:51:20.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.736+0000 7f4dbf7fe700 1 -- 172.21.15.59:0/3860093660 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4dcc00c9f0 con 0x7f4dd80fcee0 2024-01-23T08:51:20.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.736+0000 7f4ddd476700 1 -- 172.21.15.59:0/3860093660 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd81133f0 con 0x7f4dd80fcee0 2024-01-23T08:51:20.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.737+0000 7f4ddd476700 1 -- 172.21.15.59:0/3860093660 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc40052e0 con 0x7f4dd80fcee0 2024-01-23T08:51:20.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.738+0000 7f4dbf7fe700 1 -- 172.21.15.59:0/3860093660 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f4dcc00cb50 con 0x7f4dd80fcee0 2024-01-23T08:51:20.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.738+0000 7f4dbf7fe700 1 --2- 172.21.15.59:0/3860093660 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f4dc0068f70 0x7f4dc006b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:20.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.738+0000 7f4dbf7fe700 1 -- 172.21.15.59:0/3860093660 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4dcc086850 con 0x7f4dd80fcee0 2024-01-23T08:51:20.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.738+0000 7f4dd6ffd700 1 --2- 172.21.15.59:0/3860093660 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f4dc0068f70 0x7f4dc006b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:20.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.739+0000 7f4dd6ffd700 1 --2- 172.21.15.59:0/3860093660 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f4dc0068f70 0x7f4dc006b420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4dc8009fd0 tx=0x7f4dc8009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:51:20.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:20.743+0000 7f4dbf7fe700 1 -- 172.21.15.59:0/3860093660 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f4dcc052c50 con 0x7f4dd80fcee0 2024-01-23T08:51:21.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:20 smithi137 ceph-mon[35787]: Upgrade: Target is version 19.0.0-766-g0edf41a6 (unknown) 2024-01-23T08:51:21.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:20 smithi137 ceph-mon[35787]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7, digests ['quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7'] 2024-01-23T08:51:21.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:20 smithi137 ceph-mon[35787]: Upgrade: Need to upgrade myself (mgr.smithi059.bzqirf) 2024-01-23T08:51:21.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:20 smithi137 ceph-mon[35787]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7 on smithi137 2024-01-23T08:51:21.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:20 smithi137 ceph-mon[35787]: from='client.24339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:51:21.043 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:20 smithi137 ceph-mon[35787]: from='client.? 172.21.15.59:0/278273241' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-01-23T08:51:21.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:21.048+0000 7f4ddd476700 1 -- 172.21.15.59:0/3860093660 --> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4dc4000bf0 con 0x7f4dc0068f70 2024-01-23T08:51:21.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:21.049+0000 7f4dbf7fe700 1 -- 172.21.15.59:0/3860093660 <== mgr.14210 v2:172.21.15.59:6800/3219735162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f4dc4000bf0 con 0x7f4dc0068f70 2024-01-23T08:51:21.047 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-01-23T08:51:21.047 INFO:teuthology.orchestra.run.smithi059.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7", 2024-01-23T08:51:21.047 INFO:teuthology.orchestra.run.smithi059.stdout: "in_progress": true, 2024-01-23T08:51:21.047 INFO:teuthology.orchestra.run.smithi059.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-01-23T08:51:21.047 INFO:teuthology.orchestra.run.smithi059.stdout: "services_complete": [], 2024-01-23T08:51:21.047 INFO:teuthology.orchestra.run.smithi059.stdout: "progress": "0/2 daemons upgraded", 2024-01-23T08:51:21.047 INFO:teuthology.orchestra.run.smithi059.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7 image on host smithi137", 2024-01-23T08:51:21.047 INFO:teuthology.orchestra.run.smithi059.stdout: "is_paused": false 2024-01-23T08:51:21.047 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-01-23T08:51:21.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:21.052+0000 7f4ddd476700 1 -- 172.21.15.59:0/3860093660 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f4dc0068f70 msgr2=0x7f4dc006b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:21.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:21.052+0000 7f4ddd476700 1 --2- 172.21.15.59:0/3860093660 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f4dc0068f70 0x7f4dc006b420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4dc8009fd0 tx=0x7f4dc8009e30 comp rx=0 tx=0).stop 2024-01-23T08:51:21.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:21.052+0000 7f4ddd476700 1 -- 172.21.15.59:0/3860093660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4dd80fcee0 msgr2=0x7f4dd8112940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:21.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:21.052+0000 7f4ddd476700 1 --2- 172.21.15.59:0/3860093660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4dd80fcee0 0x7f4dd8112940 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc00eb70 tx=0x7f4dcc00b040 comp rx=0 tx=0).stop 2024-01-23T08:51:21.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:21.052+0000 7f4ddd476700 1 -- 172.21.15.59:0/3860093660 shutdown_connections 2024-01-23T08:51:21.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:21.052+0000 7f4ddd476700 1 --2- 172.21.15.59:0/3860093660 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f4dc0068f70 0x7f4dc006b420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:21.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:21.052+0000 7f4ddd476700 1 --2- 172.21.15.59:0/3860093660 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4dd80fbcd0 0x7f4dd81103f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:21.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:21.052+0000 7f4ddd476700 1 --2- 172.21.15.59:0/3860093660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4dd80fcee0 0x7f4dd8112940 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:21.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:21.052+0000 7f4ddd476700 1 -- 172.21.15.59:0/3860093660 >> 172.21.15.59:0/3860093660 conn(0x7f4dd80f7390 msgr2=0x7f4dd8100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:51:21.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:21.053+0000 7f4ddd476700 1 -- 172.21.15.59:0/3860093660 shutdown_connections 2024-01-23T08:51:21.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:21.053+0000 7f4ddd476700 1 -- 172.21.15.59:0/3860093660 wait complete. 2024-01-23T08:51:21.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:21 smithi059 ceph-mon[28424]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:51:21.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:21 smithi059 ceph-mon[28424]: pgmap v220: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 597 B/s wr, 4 op/s 2024-01-23T08:51:22.042 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:21 smithi137 ceph-mon[35787]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:51:22.042 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:21 smithi137 ceph-mon[35787]: pgmap v220: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 597 B/s wr, 4 op/s 2024-01-23T08:51:22.842 INFO:tasks.workunit.client.1.smithi137.stderr:Note: switching to '9db6bac5568c8be6cfd98da20bd2b62582f0776f'. 2024-01-23T08:51:22.842 INFO:tasks.workunit.client.1.smithi137.stderr: 2024-01-23T08:51:22.842 INFO:tasks.workunit.client.1.smithi137.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-01-23T08:51:22.842 INFO:tasks.workunit.client.1.smithi137.stderr:changes and commit them, and you can discard any commits you make in this 2024-01-23T08:51:22.842 INFO:tasks.workunit.client.1.smithi137.stderr:state without impacting any branches by switching back to a branch. 2024-01-23T08:51:22.842 INFO:tasks.workunit.client.1.smithi137.stderr: 2024-01-23T08:51:22.842 INFO:tasks.workunit.client.1.smithi137.stderr:If you want to create a new branch to retain commits you create, you may 2024-01-23T08:51:22.842 INFO:tasks.workunit.client.1.smithi137.stderr:do so (now or later) by using -c with the switch command. Example: 2024-01-23T08:51:22.842 INFO:tasks.workunit.client.1.smithi137.stderr: 2024-01-23T08:51:22.842 INFO:tasks.workunit.client.1.smithi137.stderr: git switch -c 2024-01-23T08:51:22.842 INFO:tasks.workunit.client.1.smithi137.stderr: 2024-01-23T08:51:22.842 INFO:tasks.workunit.client.1.smithi137.stderr:Or undo this operation with: 2024-01-23T08:51:22.842 INFO:tasks.workunit.client.1.smithi137.stderr: 2024-01-23T08:51:22.842 INFO:tasks.workunit.client.1.smithi137.stderr: git switch - 2024-01-23T08:51:22.843 INFO:tasks.workunit.client.1.smithi137.stderr: 2024-01-23T08:51:22.843 INFO:tasks.workunit.client.1.smithi137.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-01-23T08:51:22.843 INFO:tasks.workunit.client.1.smithi137.stderr: 2024-01-23T08:51:22.843 INFO:tasks.workunit.client.1.smithi137.stderr:HEAD is now at 9db6bac5568 Merge branch 'pr-55233' into wip-yuri8-testing-2024-01-18-0823 2024-01-23T08:51:22.847 DEBUG:teuthology.orchestra.run.smithi137:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-01-23T08:51:22.909 INFO:tasks.workunit.client.1.smithi137.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-01-23T08:51:22.911 INFO:tasks.workunit.client.1.smithi137.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-01-23T08:51:22.911 INFO:tasks.workunit.client.1.smithi137.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-01-23T08:51:22.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:22 smithi059 ceph-mon[28424]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:51:23.042 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:22 smithi137 ceph-mon[35787]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:51:23.686 INFO:tasks.workunit.client.1.smithi137.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-01-23T08:51:23.751 INFO:tasks.workunit.client.1.smithi137.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-01-23T08:51:23.793 INFO:tasks.workunit.client.1.smithi137.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-01-23T08:51:23.794 INFO:tasks.workunit.client.1.smithi137.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-01-23T08:51:23.794 INFO:tasks.workunit.client.1.smithi137.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-01-23T08:51:23.834 INFO:tasks.workunit.client.1.smithi137.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-01-23T08:51:23.837 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-23T08:51:23.837 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-01-23T08:51:23.893 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-01-23T08:51:23.895 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-01-23T08:51:23.896 DEBUG:teuthology.orchestra.run.smithi137:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9db6bac5568c8be6cfd98da20bd2b62582f0776f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-01-23T08:51:23.960 INFO:tasks.workunit.client.1.smithi137.stderr:+ mkdir -p fsstress 2024-01-23T08:51:23.962 INFO:tasks.workunit.client.1.smithi137.stderr:+ pushd fsstress 2024-01-23T08:51:23.962 INFO:tasks.workunit.client.1.smithi137.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-01-23T08:51:23.962 INFO:tasks.workunit.client.1.smithi137.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-01-23T08:51:23.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:23 smithi059 ceph-mon[28424]: pgmap v221: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 5 op/s 2024-01-23T08:51:24.042 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:23 smithi137 ceph-mon[35787]: pgmap v221: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 5 op/s 2024-01-23T08:51:24.850 INFO:tasks.workunit.client.1.smithi137.stderr:+ tar xzf ltp-full.tgz 2024-01-23T08:51:25.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:25 smithi059 ceph-mon[28424]: pgmap v222: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 3 op/s 2024-01-23T08:51:25.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:25 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:51:26.042 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:25 smithi137 ceph-mon[35787]: pgmap v222: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 3 op/s 2024-01-23T08:51:26.042 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:25 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:51:27.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:27 smithi059 ceph-mon[28424]: pgmap v223: 65 pgs: 65 active+clean; 1.9 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 111 KiB/s wr, 5 op/s 2024-01-23T08:51:28.042 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:27 smithi137 ceph-mon[35787]: pgmap v223: 65 pgs: 65 active+clean; 1.9 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 111 KiB/s wr, 5 op/s 2024-01-23T08:51:28.684 INFO:tasks.workunit.client.0.smithi059.stderr:Note: switching to '9db6bac5568c8be6cfd98da20bd2b62582f0776f'. 2024-01-23T08:51:28.684 INFO:tasks.workunit.client.0.smithi059.stderr: 2024-01-23T08:51:28.684 INFO:tasks.workunit.client.0.smithi059.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-01-23T08:51:28.684 INFO:tasks.workunit.client.0.smithi059.stderr:changes and commit them, and you can discard any commits you make in this 2024-01-23T08:51:28.684 INFO:tasks.workunit.client.0.smithi059.stderr:state without impacting any branches by switching back to a branch. 2024-01-23T08:51:28.684 INFO:tasks.workunit.client.0.smithi059.stderr: 2024-01-23T08:51:28.684 INFO:tasks.workunit.client.0.smithi059.stderr:If you want to create a new branch to retain commits you create, you may 2024-01-23T08:51:28.684 INFO:tasks.workunit.client.0.smithi059.stderr:do so (now or later) by using -c with the switch command. Example: 2024-01-23T08:51:28.684 INFO:tasks.workunit.client.0.smithi059.stderr: 2024-01-23T08:51:28.684 INFO:tasks.workunit.client.0.smithi059.stderr: git switch -c 2024-01-23T08:51:28.684 INFO:tasks.workunit.client.0.smithi059.stderr: 2024-01-23T08:51:28.685 INFO:tasks.workunit.client.0.smithi059.stderr:Or undo this operation with: 2024-01-23T08:51:28.685 INFO:tasks.workunit.client.0.smithi059.stderr: 2024-01-23T08:51:28.685 INFO:tasks.workunit.client.0.smithi059.stderr: git switch - 2024-01-23T08:51:28.685 INFO:tasks.workunit.client.0.smithi059.stderr: 2024-01-23T08:51:28.685 INFO:tasks.workunit.client.0.smithi059.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-01-23T08:51:28.685 INFO:tasks.workunit.client.0.smithi059.stderr: 2024-01-23T08:51:28.685 INFO:tasks.workunit.client.0.smithi059.stderr:HEAD is now at 9db6bac5568 Merge branch 'pr-55233' into wip-yuri8-testing-2024-01-18-0823 2024-01-23T08:51:28.689 DEBUG:teuthology.orchestra.run.smithi059:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-01-23T08:51:28.713 INFO:tasks.workunit.client.0.smithi059.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-01-23T08:51:28.715 INFO:tasks.workunit.client.0.smithi059.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-01-23T08:51:28.715 INFO:tasks.workunit.client.0.smithi059.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-01-23T08:51:29.684 INFO:tasks.workunit.client.0.smithi059.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-01-23T08:51:29.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:29 smithi059 ceph-mon[28424]: pgmap v224: 65 pgs: 65 active+clean; 1.9 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 111 KiB/s wr, 4 op/s 2024-01-23T08:51:29.979 INFO:tasks.workunit.client.0.smithi059.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-01-23T08:51:30.042 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:29 smithi137 ceph-mon[35787]: pgmap v224: 65 pgs: 65 active+clean; 1.9 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 111 KiB/s wr, 4 op/s 2024-01-23T08:51:30.107 INFO:tasks.workunit.client.0.smithi059.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-01-23T08:51:30.109 INFO:tasks.workunit.client.0.smithi059.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-01-23T08:51:30.109 INFO:tasks.workunit.client.0.smithi059.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-01-23T08:51:30.165 INFO:tasks.workunit.client.0.smithi059.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-01-23T08:51:30.168 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-23T08:51:30.168 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-01-23T08:51:30.223 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-01-23T08:51:30.224 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-01-23T08:51:30.224 DEBUG:teuthology.orchestra.run.smithi059:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9db6bac5568c8be6cfd98da20bd2b62582f0776f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-01-23T08:51:30.283 INFO:tasks.workunit.client.0.smithi059.stderr:+ mkdir -p fsstress 2024-01-23T08:51:30.285 INFO:tasks.workunit.client.0.smithi059.stderr:+ pushd fsstress 2024-01-23T08:51:30.285 INFO:tasks.workunit.client.0.smithi059.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-01-23T08:51:30.286 INFO:tasks.workunit.client.0.smithi059.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-01-23T08:51:31.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:31 smithi059 ceph-mon[28424]: pgmap v225: 65 pgs: 65 active+clean; 1.9 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 111 KiB/s wr, 3 op/s 2024-01-23T08:51:31.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:31 smithi137 ceph-mon[35787]: pgmap v225: 65 pgs: 65 active+clean; 1.9 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 111 KiB/s wr, 3 op/s 2024-01-23T08:51:31.691 INFO:tasks.workunit.client.0.smithi059.stderr:+ tar xzf ltp-full.tgz 2024-01-23T08:51:33.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:33 smithi059 ceph-mon[28424]: pgmap v226: 65 pgs: 65 active+clean; 28 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 909 KiB/s rd, 2.4 MiB/s wr, 183 op/s 2024-01-23T08:51:33.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:33 smithi137 ceph-mon[35787]: pgmap v226: 65 pgs: 65 active+clean; 28 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 909 KiB/s rd, 2.4 MiB/s wr, 183 op/s 2024-01-23T08:51:35.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:35 smithi059 ceph-mon[28424]: pgmap v227: 65 pgs: 65 active+clean; 49 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 4.3 MiB/s wr, 409 op/s 2024-01-23T08:51:35.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:35 smithi137 ceph-mon[35787]: pgmap v227: 65 pgs: 65 active+clean; 49 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 4.3 MiB/s wr, 409 op/s 2024-01-23T08:51:37.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:37 smithi059 ceph-mon[28424]: pgmap v228: 65 pgs: 65 active+clean; 57 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 1.7 MiB/s rd, 5.0 MiB/s wr, 435 op/s 2024-01-23T08:51:37.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:37 smithi137 ceph-mon[35787]: pgmap v228: 65 pgs: 65 active+clean; 57 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 1.7 MiB/s rd, 5.0 MiB/s wr, 435 op/s 2024-01-23T08:51:39.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:39 smithi059 ceph-mon[28424]: pgmap v229: 65 pgs: 65 active+clean; 72 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 6.1 MiB/s wr, 510 op/s 2024-01-23T08:51:39.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:39 smithi059 ceph-mon[28424]: fsmap cephfs:2 {0=cephfs.smithi059.twpxfz=up:active,1=cephfs.smithi137.mgacmp=up:active} 2 up:standby-replay 2024-01-23T08:51:39.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:39 smithi137 ceph-mon[35787]: pgmap v229: 65 pgs: 65 active+clean; 72 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 6.1 MiB/s wr, 510 op/s 2024-01-23T08:51:39.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:39 smithi137 ceph-mon[35787]: fsmap cephfs:2 {0=cephfs.smithi059.twpxfz=up:active,1=cephfs.smithi137.mgacmp=up:active} 2 up:standby-replay 2024-01-23T08:51:41.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:41 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:51:41.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:41 smithi059 ceph-mon[28424]: pgmap v230: 65 pgs: 65 active+clean; 72 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 6.1 MiB/s wr, 509 op/s 2024-01-23T08:51:41.542 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:41 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-23T08:51:41.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:41 smithi137 ceph-mon[35787]: pgmap v230: 65 pgs: 65 active+clean; 72 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 6.1 MiB/s wr, 509 op/s 2024-01-23T08:51:43.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:43 smithi059 ceph-mon[28424]: pgmap v231: 65 pgs: 65 active+clean; 87 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 3.1 MiB/s rd, 7.5 MiB/s wr, 548 op/s 2024-01-23T08:51:43.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:43 smithi137 ceph-mon[35787]: pgmap v231: 65 pgs: 65 active+clean; 87 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 3.1 MiB/s rd, 7.5 MiB/s wr, 548 op/s 2024-01-23T08:51:45.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:45 smithi059 ceph-mon[28424]: pgmap v232: 65 pgs: 65 active+clean; 95 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.6 MiB/s rd, 5.8 MiB/s wr, 369 op/s 2024-01-23T08:51:45.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:45 smithi137 ceph-mon[35787]: pgmap v232: 65 pgs: 65 active+clean; 95 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.6 MiB/s rd, 5.8 MiB/s wr, 369 op/s 2024-01-23T08:51:47.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:47 smithi059 ceph-mon[28424]: pgmap v233: 65 pgs: 65 active+clean; 109 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 2.6 MiB/s rd, 5.2 MiB/s wr, 337 op/s 2024-01-23T08:51:47.792 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:47 smithi137 ceph-mon[35787]: pgmap v233: 65 pgs: 65 active+clean; 109 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 2.6 MiB/s rd, 5.2 MiB/s wr, 337 op/s 2024-01-23T08:51:49.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:49 smithi137 ceph-mon[35787]: pgmap v234: 65 pgs: 65 active+clean; 137 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.5 MiB/s rd, 7.1 MiB/s wr, 696 op/s 2024-01-23T08:51:49.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:49 smithi137 ceph-mon[35787]: Upgrade: Updating mgr.smithi137.ooirrh 2024-01-23T08:51:49.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:49 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:51:49.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:49 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.ooirrh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-23T08:51:49.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:49 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-23T08:51:49.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:49 smithi137 ceph-mon[35787]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:51:49.793 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:49 smithi137 ceph-mon[35787]: Deploying daemon mgr.smithi137.ooirrh on smithi137 2024-01-23T08:51:49.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:49 smithi059 ceph-mon[28424]: pgmap v234: 65 pgs: 65 active+clean; 137 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.5 MiB/s rd, 7.1 MiB/s wr, 696 op/s 2024-01-23T08:51:49.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:49 smithi059 ceph-mon[28424]: Upgrade: Updating mgr.smithi137.ooirrh 2024-01-23T08:51:49.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:49 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' 2024-01-23T08:51:49.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:49 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.ooirrh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-23T08:51:49.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:49 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-23T08:51:49.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:49 smithi059 ceph-mon[28424]: from='mgr.14210 172.21.15.59:0/436818147' entity='mgr.smithi059.bzqirf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-23T08:51:49.971 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:49 smithi059 ceph-mon[28424]: Deploying daemon mgr.smithi137.ooirrh on smithi137 2024-01-23T08:51:51.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.168+0000 7fbd1502d700 1 -- 172.21.15.59:0/17060973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd100fb0b0 msgr2=0x7fbd100fb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:51.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.168+0000 7fbd1502d700 1 --2- 172.21.15.59:0/17060973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd100fb0b0 0x7fbd100fb4c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fbd00004940 tx=0x7fbd00009e30 comp rx=0 tx=0).stop 2024-01-23T08:51:51.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.168+0000 7fbd1502d700 1 -- 172.21.15.59:0/17060973 shutdown_connections 2024-01-23T08:51:51.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.168+0000 7fbd1502d700 1 --2- 172.21.15.59:0/17060973 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd100fe140 0x7fbd100fe590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:51.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.168+0000 7fbd1502d700 1 --2- 172.21.15.59:0/17060973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd100fb0b0 0x7fbd100fb4c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:51.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.168+0000 7fbd1502d700 1 -- 172.21.15.59:0/17060973 >> 172.21.15.59:0/17060973 conn(0x7fbd100f6b30 msgr2=0x7fbd100f8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:51:51.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.168+0000 7fbd1502d700 1 -- 172.21.15.59:0/17060973 shutdown_connections 2024-01-23T08:51:51.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.168+0000 7fbd1502d700 1 -- 172.21.15.59:0/17060973 wait complete. 2024-01-23T08:51:51.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.169+0000 7fbd1502d700 1 Processor -- start 2024-01-23T08:51:51.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.169+0000 7fbd1502d700 1 -- start start 2024-01-23T08:51:51.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.169+0000 7fbd1502d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd100fb0b0 0x7fbd1010d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:51.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.169+0000 7fbd1502d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd100fe140 0x7fbd1010de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:51.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.169+0000 7fbd1502d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1010e400 con 0x7fbd100fe140 2024-01-23T08:51:51.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.169+0000 7fbd1502d700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1010eca0 con 0x7fbd100fb0b0 2024-01-23T08:51:51.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.169+0000 7fbd0e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd100fe140 0x7fbd1010de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:51.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.169+0000 7fbd0e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd100fe140 0x7fbd1010de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60154/0 (socket says 172.21.15.59:60154) 2024-01-23T08:51:51.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.169+0000 7fbd0e59c700 1 -- 172.21.15.59:0/1051010977 learned_addr learned my addr 172.21.15.59:0/1051010977 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:51:51.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.169+0000 7fbd0ed9d700 1 --2- 172.21.15.59:0/1051010977 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd100fb0b0 0x7fbd1010d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:51.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.170+0000 7fbd0e59c700 1 -- 172.21.15.59:0/1051010977 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd100fb0b0 msgr2=0x7fbd1010d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:51.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.170+0000 7fbd0e59c700 1 --2- 172.21.15.59:0/1051010977 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd100fb0b0 0x7fbd1010d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:51.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.170+0000 7fbd0e59c700 1 -- 172.21.15.59:0/1051010977 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd00005040 con 0x7fbd100fe140 2024-01-23T08:51:51.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.170+0000 7fbd0ed9d700 1 --2- 172.21.15.59:0/1051010977 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd100fb0b0 0x7fbd1010d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-23T08:51:51.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.170+0000 7fbd0e59c700 1 --2- 172.21.15.59:0/1051010977 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd100fe140 0x7fbd1010de00 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fbd0400eb40 tx=0x7fbd0400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:51:51.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.170+0000 7fbcf77fe700 1 -- 172.21.15.59:0/1051010977 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd04007cc0 con 0x7fbd100fe140 2024-01-23T08:51:51.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.170+0000 7fbd1502d700 1 -- 172.21.15.59:0/1051010977 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd1010ef20 con 0x7fbd100fe140 2024-01-23T08:51:51.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.170+0000 7fbcf77fe700 1 -- 172.21.15.59:0/1051010977 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd0400f040 con 0x7fbd100fe140 2024-01-23T08:51:51.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.170+0000 7fbcf77fe700 1 -- 172.21.15.59:0/1051010977 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd0400c990 con 0x7fbd100fe140 2024-01-23T08:51:51.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.170+0000 7fbd1502d700 1 -- 172.21.15.59:0/1051010977 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd1010f460 con 0x7fbd100fe140 2024-01-23T08:51:51.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.171+0000 7fbd1502d700 1 -- 172.21.15.59:0/1051010977 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd10107700 con 0x7fbd100fe140 2024-01-23T08:51:51.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.172+0000 7fbcf77fe700 1 -- 172.21.15.59:0/1051010977 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fbd0400cc00 con 0x7fbd100fe140 2024-01-23T08:51:51.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.172+0000 7fbcf77fe700 1 --2- 172.21.15.59:0/1051010977 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbcf8068f60 0x7fbcf806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:51.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.172+0000 7fbcf77fe700 1 -- 172.21.15.59:0/1051010977 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fbd040869e0 con 0x7fbd100fe140 2024-01-23T08:51:51.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.172+0000 7fbd0ed9d700 1 --2- 172.21.15.59:0/1051010977 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbcf8068f60 0x7fbcf806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:51.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.173+0000 7fbd0ed9d700 1 --2- 172.21.15.59:0/1051010977 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbcf8068f60 0x7fbcf806b410 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbd00000c00 tx=0x7fbd00009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:51:51.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.177+0000 7fbcf77fe700 1 -- 172.21.15.59:0/1051010977 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fbd0404da00 con 0x7fbd100fe140 2024-01-23T08:51:51.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:51 smithi137 ceph-mon[35787]: pgmap v235: 65 pgs: 65 active+clean; 137 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 5.8 MiB/s wr, 619 op/s 2024-01-23T08:51:51.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.396+0000 7fbd1502d700 1 -- 172.21.15.59:0/1051010977 --> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd1010f680 con 0x7fbcf8068f60 2024-01-23T08:51:51.393 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:51 smithi059 ceph-mon[28424]: pgmap v235: 65 pgs: 65 active+clean; 137 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 5.8 MiB/s wr, 619 op/s 2024-01-23T08:51:51.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.397+0000 7fbcf77fe700 1 -- 172.21.15.59:0/1051010977 <== mgr.14210 v2:172.21.15.59:6800/3219735162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fbd1010f680 con 0x7fbcf8068f60 2024-01-23T08:51:51.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.399+0000 7fbd1502d700 1 -- 172.21.15.59:0/1051010977 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbcf8068f60 msgr2=0x7fbcf806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:51.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.399+0000 7fbd1502d700 1 --2- 172.21.15.59:0/1051010977 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbcf8068f60 0x7fbcf806b410 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbd00000c00 tx=0x7fbd00009e30 comp rx=0 tx=0).stop 2024-01-23T08:51:51.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.399+0000 7fbd1502d700 1 -- 172.21.15.59:0/1051010977 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd100fe140 msgr2=0x7fbd1010de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:51.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.399+0000 7fbd1502d700 1 --2- 172.21.15.59:0/1051010977 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd100fe140 0x7fbd1010de00 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fbd0400eb40 tx=0x7fbd0400b040 comp rx=0 tx=0).stop 2024-01-23T08:51:51.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.399+0000 7fbd1502d700 1 -- 172.21.15.59:0/1051010977 shutdown_connections 2024-01-23T08:51:51.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.399+0000 7fbd1502d700 1 --2- 172.21.15.59:0/1051010977 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fbcf8068f60 0x7fbcf806b410 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:51.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.399+0000 7fbd1502d700 1 --2- 172.21.15.59:0/1051010977 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd100fb0b0 0x7fbd1010d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:51.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.399+0000 7fbd1502d700 1 --2- 172.21.15.59:0/1051010977 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd100fe140 0x7fbd1010de00 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:51.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.399+0000 7fbd1502d700 1 -- 172.21.15.59:0/1051010977 >> 172.21.15.59:0/1051010977 conn(0x7fbd100f6b30 msgr2=0x7fbd10101c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:51:51.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.399+0000 7fbd1502d700 1 -- 172.21.15.59:0/1051010977 shutdown_connections 2024-01-23T08:51:51.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.399+0000 7fbd1502d700 1 -- 172.21.15.59:0/1051010977 wait complete. 2024-01-23T08:51:51.407 INFO:teuthology.orchestra.run.smithi059.stdout:true 2024-01-23T08:51:51.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.508+0000 7fef36750700 1 -- 172.21.15.59:0/4285962580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef300fff50 msgr2=0x7fef301023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:51.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.508+0000 7fef36750700 1 --2- 172.21.15.59:0/4285962580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef300fff50 0x7fef301023b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fef24004940 tx=0x7fef24009e30 comp rx=0 tx=0).stop 2024-01-23T08:51:51.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.508+0000 7fef36750700 1 -- 172.21.15.59:0/4285962580 shutdown_connections 2024-01-23T08:51:51.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.508+0000 7fef36750700 1 --2- 172.21.15.59:0/4285962580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef300fff50 0x7fef301023b0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:51.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.508+0000 7fef36750700 1 --2- 172.21.15.59:0/4285962580 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef30066270 0x7fef300ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:51.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.508+0000 7fef36750700 1 -- 172.21.15.59:0/4285962580 >> 172.21.15.59:0/4285962580 conn(0x7fef300f7370 msgr2=0x7fef300f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:51:51.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.508+0000 7fef36750700 1 -- 172.21.15.59:0/4285962580 shutdown_connections 2024-01-23T08:51:51.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.508+0000 7fef36750700 1 -- 172.21.15.59:0/4285962580 wait complete. 2024-01-23T08:51:51.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.508+0000 7fef36750700 1 Processor -- start 2024-01-23T08:51:51.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.509+0000 7fef36750700 1 -- start start 2024-01-23T08:51:51.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.509+0000 7fef36750700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef30066270 0x7fef300fd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:51.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.509+0000 7fef36750700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef300fff50 0x7fef300fb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:51.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.509+0000 7fef36750700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef300fbe80 con 0x7fef30066270 2024-01-23T08:51:51.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.509+0000 7fef36750700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef300fbff0 con 0x7fef300fff50 2024-01-23T08:51:51.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.509+0000 7fef2ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef30066270 0x7fef300fd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:51.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.509+0000 7fef2ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef30066270 0x7fef300fd400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60182/0 (socket says 172.21.15.59:60182) 2024-01-23T08:51:51.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.509+0000 7fef2f7fe700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef300fff50 0x7fef300fb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:51.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.509+0000 7fef2ffff700 1 -- 172.21.15.59:0/1857782374 learned_addr learned my addr 172.21.15.59:0/1857782374 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:51:51.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.509+0000 7fef2f7fe700 1 -- 172.21.15.59:0/1857782374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef30066270 msgr2=0x7fef300fd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:51.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.509+0000 7fef2f7fe700 1 --2- 172.21.15.59:0/1857782374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef30066270 0x7fef300fd400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:51.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.509+0000 7fef2f7fe700 1 -- 172.21.15.59:0/1857782374 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef24005040 con 0x7fef300fff50 2024-01-23T08:51:51.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.509+0000 7fef2ffff700 1 --2- 172.21.15.59:0/1857782374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef30066270 0x7fef300fd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-23T08:51:51.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.510+0000 7fef2f7fe700 1 --2- 172.21.15.59:0/1857782374 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef300fff50 0x7fef300fb880 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fef2400b0d0 tx=0x7fef24009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:51:51.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.510+0000 7fef2cff9700 1 -- 172.21.15.59:0/1857782374 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef2400e070 con 0x7fef300fff50 2024-01-23T08:51:51.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.510+0000 7fef36750700 1 -- 172.21.15.59:0/1857782374 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef300fc270 con 0x7fef300fff50 2024-01-23T08:51:51.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.510+0000 7fef2cff9700 1 -- 172.21.15.59:0/1857782374 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef2400b410 con 0x7fef300fff50 2024-01-23T08:51:51.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.510+0000 7fef2cff9700 1 -- 172.21.15.59:0/1857782374 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef24012600 con 0x7fef300fff50 2024-01-23T08:51:51.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.510+0000 7fef36750700 1 -- 172.21.15.59:0/1857782374 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef30115820 con 0x7fef300fff50 2024-01-23T08:51:51.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.511+0000 7fef36750700 1 -- 172.21.15.59:0/1857782374 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef301076d0 con 0x7fef300fff50 2024-01-23T08:51:51.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.511+0000 7fef2cff9700 1 -- 172.21.15.59:0/1857782374 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fef2400b9f0 con 0x7fef300fff50 2024-01-23T08:51:51.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.512+0000 7fef2cff9700 1 --2- 172.21.15.59:0/1857782374 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fef18068f60 0x7fef1806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:51.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.512+0000 7fef2cff9700 1 -- 172.21.15.59:0/1857782374 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fef240877a0 con 0x7fef300fff50 2024-01-23T08:51:51.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.512+0000 7fef2ffff700 1 --2- 172.21.15.59:0/1857782374 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fef18068f60 0x7fef1806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:51.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.512+0000 7fef2ffff700 1 --2- 172.21.15.59:0/1857782374 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fef18068f60 0x7fef1806b410 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fef20004fd0 tx=0x7fef2000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:51:51.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.516+0000 7fef2cff9700 1 -- 172.21.15.59:0/1857782374 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fef24057690 con 0x7fef300fff50 2024-01-23T08:51:51.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.784+0000 7fef36750700 1 -- 172.21.15.59:0/1857782374 --> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fef3005e260 con 0x7fef18068f60 2024-01-23T08:51:51.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.785+0000 7fef2cff9700 1 -- 172.21.15.59:0/1857782374 <== mgr.14210 v2:172.21.15.59:6800/3219735162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fef3005e260 con 0x7fef18068f60 2024-01-23T08:51:51.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.787+0000 7fef36750700 1 -- 172.21.15.59:0/1857782374 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fef18068f60 msgr2=0x7fef1806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:51.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.787+0000 7fef36750700 1 --2- 172.21.15.59:0/1857782374 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fef18068f60 0x7fef1806b410 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fef20004fd0 tx=0x7fef2000b040 comp rx=0 tx=0).stop 2024-01-23T08:51:51.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.787+0000 7fef36750700 1 -- 172.21.15.59:0/1857782374 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef300fff50 msgr2=0x7fef300fb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:51.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.787+0000 7fef36750700 1 --2- 172.21.15.59:0/1857782374 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef300fff50 0x7fef300fb880 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fef2400b0d0 tx=0x7fef24009e30 comp rx=0 tx=0).stop 2024-01-23T08:51:51.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.787+0000 7fef36750700 1 -- 172.21.15.59:0/1857782374 shutdown_connections 2024-01-23T08:51:51.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.787+0000 7fef36750700 1 --2- 172.21.15.59:0/1857782374 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fef18068f60 0x7fef1806b410 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:51.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.787+0000 7fef36750700 1 --2- 172.21.15.59:0/1857782374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef30066270 0x7fef300fd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:51.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.787+0000 7fef36750700 1 --2- 172.21.15.59:0/1857782374 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef300fff50 0x7fef300fb880 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:51.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.787+0000 7fef36750700 1 -- 172.21.15.59:0/1857782374 >> 172.21.15.59:0/1857782374 conn(0x7fef300f7370 msgr2=0x7fef300f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:51:51.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.788+0000 7fef36750700 1 -- 172.21.15.59:0/1857782374 shutdown_connections 2024-01-23T08:51:51.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.788+0000 7fef36750700 1 -- 172.21.15.59:0/1857782374 wait complete. 2024-01-23T08:51:51.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.899+0000 7fcb212a3700 1 -- 172.21.15.59:0/3168528561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb1c0fd910 msgr2=0x7fcb1c0ffcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:51.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.900+0000 7fcb212a3700 1 --2- 172.21.15.59:0/3168528561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb1c0fd910 0x7fcb1c0ffcf0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c004940 tx=0x7fcb0c009e30 comp rx=0 tx=0).stop 2024-01-23T08:51:51.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.900+0000 7fcb212a3700 1 -- 172.21.15.59:0/3168528561 shutdown_connections 2024-01-23T08:51:51.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.900+0000 7fcb212a3700 1 --2- 172.21.15.59:0/3168528561 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb1c100230 0x7fcb1c102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:51.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.900+0000 7fcb212a3700 1 --2- 172.21.15.59:0/3168528561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb1c0fd910 0x7fcb1c0ffcf0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:51.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.900+0000 7fcb212a3700 1 -- 172.21.15.59:0/3168528561 >> 172.21.15.59:0/3168528561 conn(0x7fcb1c0f7370 msgr2=0x7fcb1c0f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:51:51.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.900+0000 7fcb212a3700 1 -- 172.21.15.59:0/3168528561 shutdown_connections 2024-01-23T08:51:51.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.900+0000 7fcb212a3700 1 -- 172.21.15.59:0/3168528561 wait complete. 2024-01-23T08:51:51.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.901+0000 7fcb212a3700 1 Processor -- start 2024-01-23T08:51:51.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.901+0000 7fcb212a3700 1 -- start start 2024-01-23T08:51:51.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.901+0000 7fcb212a3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb1c0fd910 0x7fcb1c10d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:51.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.901+0000 7fcb212a3700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb1c100230 0x7fcb1c10dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:51.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.901+0000 7fcb212a3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c10e330 con 0x7fcb1c0fd910 2024-01-23T08:51:51.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.901+0000 7fcb212a3700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c10ebd0 con 0x7fcb1c100230 2024-01-23T08:51:51.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.901+0000 7fcb1affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb1c0fd910 0x7fcb1c10d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:51.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.901+0000 7fcb1affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb1c0fd910 0x7fcb1c10d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60196/0 (socket says 172.21.15.59:60196) 2024-01-23T08:51:51.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.901+0000 7fcb1affd700 1 -- 172.21.15.59:0/1374594586 learned_addr learned my addr 172.21.15.59:0/1374594586 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:51:51.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.901+0000 7fcb1a7fc700 1 --2- 172.21.15.59:0/1374594586 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb1c100230 0x7fcb1c10dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:51.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.901+0000 7fcb1affd700 1 -- 172.21.15.59:0/1374594586 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb1c100230 msgr2=0x7fcb1c10dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:51.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.901+0000 7fcb1affd700 1 --2- 172.21.15.59:0/1374594586 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb1c100230 0x7fcb1c10dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:51.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.902+0000 7fcb1affd700 1 -- 172.21.15.59:0/1374594586 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0c005040 con 0x7fcb1c0fd910 2024-01-23T08:51:51.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.902+0000 7fcb1affd700 1 --2- 172.21.15.59:0/1374594586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb1c0fd910 0x7fcb1c10d7f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c009360 tx=0x7fcb0c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:51:51.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.902+0000 7fcb037fe700 1 -- 172.21.15.59:0/1374594586 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb0c00e040 con 0x7fcb1c0fd910 2024-01-23T08:51:51.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.902+0000 7fcb037fe700 1 -- 172.21.15.59:0/1374594586 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb0c00bc50 con 0x7fcb1c0fd910 2024-01-23T08:51:51.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.902+0000 7fcb212a3700 1 -- 172.21.15.59:0/1374594586 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb1c10ee50 con 0x7fcb1c0fd910 2024-01-23T08:51:51.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.902+0000 7fcb037fe700 1 -- 172.21.15.59:0/1374594586 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb0c01c3f0 con 0x7fcb1c0fd910 2024-01-23T08:51:51.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.902+0000 7fcb212a3700 1 -- 172.21.15.59:0/1374594586 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb1c10f390 con 0x7fcb1c0fd910 2024-01-23T08:51:51.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.903+0000 7fcb212a3700 1 -- 172.21.15.59:0/1374594586 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb1c1076d0 con 0x7fcb1c0fd910 2024-01-23T08:51:51.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.904+0000 7fcb037fe700 1 -- 172.21.15.59:0/1374594586 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fcb0c013420 con 0x7fcb1c0fd910 2024-01-23T08:51:51.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.905+0000 7fcb037fe700 1 --2- 172.21.15.59:0/1374594586 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fcb04068f60 0x7fcb0406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:51.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.905+0000 7fcb037fe700 1 -- 172.21.15.59:0/1374594586 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fcb0c088060 con 0x7fcb1c0fd910 2024-01-23T08:51:51.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.905+0000 7fcb1a7fc700 1 --2- 172.21.15.59:0/1374594586 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fcb04068f60 0x7fcb0406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:51.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.905+0000 7fcb1a7fc700 1 --2- 172.21.15.59:0/1374594586 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fcb04068f60 0x7fcb0406b410 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fcb10004f40 tx=0x7fcb1000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:51:51.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:51.913+0000 7fcb037fe700 1 -- 172.21.15.59:0/1374594586 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fcb0c057f70 con 0x7fcb1c0fd910 2024-01-23T08:51:52.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.151+0000 7fcb212a3700 1 -- 172.21.15.59:0/1374594586 --> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb1c02b240 con 0x7fcb04068f60 2024-01-23T08:51:52.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.161+0000 7fcb037fe700 1 -- 172.21.15.59:0/1374594586 <== mgr.14210 v2:172.21.15.59:6800/3219735162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7fcb1c02b240 con 0x7fcb04068f60 2024-01-23T08:51:52.158 INFO:teuthology.orchestra.run.smithi059.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-01-23T08:51:52.158 INFO:teuthology.orchestra.run.smithi059.stdout:alertmanager.smithi059 smithi059 *:9093,9094 running (5m) 86s ago 7m 25.7M - 0.25.0 c8568f914cd2 1cc540895329 2024-01-23T08:51:52.159 INFO:teuthology.orchestra.run.smithi059.stdout:crash.smithi059 smithi059 running (7m) 86s ago 7m 6656k - 17.2.7-397-gb824e5be 61b360e6ff65 12d41238ee1c 2024-01-23T08:51:52.159 INFO:teuthology.orchestra.run.smithi059.stdout:crash.smithi137 smithi137 running (6m) 91s ago 6m 6665k - 17.2.7-397-gb824e5be 61b360e6ff65 62f889b4ea0f 2024-01-23T08:51:52.159 INFO:teuthology.orchestra.run.smithi059.stdout:grafana.smithi059 smithi059 *:3000 running (5m) 86s ago 7m 88.1M - 9.4.7 954c08fa6188 f2e5dfb17ced 2024-01-23T08:51:52.159 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi059.lsxsmm smithi059 running (112s) 86s ago 110s 11.5M - 17.2.7-397-gb824e5be 61b360e6ff65 f35175705b99 2024-01-23T08:51:52.159 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi059.twpxfz smithi059 running (104s) 86s ago 104s 14.8M - 17.2.7-397-gb824e5be 61b360e6ff65 bbe0beb326d2 2024-01-23T08:51:52.159 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi137.mgacmp smithi137 running (108s) 91s ago 108s 14.5M - 17.2.7-397-gb824e5be 61b360e6ff65 57bc18125785 2024-01-23T08:51:52.159 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi137.scidwx smithi137 running (102s) 91s ago 101s 15.5M - 17.2.7-397-gb824e5be 61b360e6ff65 79e1d1658276 2024-01-23T08:51:52.159 INFO:teuthology.orchestra.run.smithi059.stdout:mgr.smithi059.bzqirf smithi059 *:9283 running (8m) 86s ago 8m 481M - 17.2.7-397-gb824e5be 61b360e6ff65 88cf77d2510a 2024-01-23T08:51:52.159 INFO:teuthology.orchestra.run.smithi059.stdout:mgr.smithi137.ooirrh smithi137 *:8443,9283 running (6m) 91s ago 6m 431M - 17.2.7-397-gb824e5be 61b360e6ff65 5af177b46b24 2024-01-23T08:51:52.159 INFO:teuthology.orchestra.run.smithi059.stdout:mon.smithi059 smithi059 running (8m) 86s ago 8m 46.5M 2048M 17.2.7-397-gb824e5be 61b360e6ff65 810a40012e10 2024-01-23T08:51:52.159 INFO:teuthology.orchestra.run.smithi059.stdout:mon.smithi137 smithi137 running (6m) 91s ago 6m 36.3M 2048M 17.2.7-397-gb824e5be 61b360e6ff65 e1d4c945bf49 2024-01-23T08:51:52.159 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter.smithi059 smithi059 *:9100 running (7m) 86s ago 7m 15.9M - 1.5.0 0da6a335fe13 89ec78f972e4 2024-01-23T08:51:52.159 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter.smithi137 smithi137 *:9100 running (6m) 91s ago 6m 15.1M - 1.5.0 0da6a335fe13 1b38a323818a 2024-01-23T08:51:52.160 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 smithi059 running (5m) 86s ago 5m 46.6M 4289M 17.2.7-397-gb824e5be 61b360e6ff65 11074a2f09ab 2024-01-23T08:51:52.160 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 smithi059 running (5m) 86s ago 5m 48.8M 4289M 17.2.7-397-gb824e5be 61b360e6ff65 efe49a487df2 2024-01-23T08:51:52.160 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 smithi059 running (4m) 86s ago 4m 47.4M 4289M 17.2.7-397-gb824e5be 61b360e6ff65 0e2f974c19f0 2024-01-23T08:51:52.160 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 smithi137 running (4m) 91s ago 4m 44.4M 5312M 17.2.7-397-gb824e5be 61b360e6ff65 a7de37e3a235 2024-01-23T08:51:52.160 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 smithi137 running (3m) 91s ago 3m 42.8M 5312M 17.2.7-397-gb824e5be 61b360e6ff65 017510d7e50d 2024-01-23T08:51:52.160 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 smithi137 running (3m) 91s ago 3m 42.5M 5312M 17.2.7-397-gb824e5be 61b360e6ff65 c37971a0dbb9 2024-01-23T08:51:52.160 INFO:teuthology.orchestra.run.smithi059.stdout:prometheus.smithi059 smithi059 *:9095 running (5m) 86s ago 6m 52.1M - 2.43.0 a07b618ecd1d 21db8bea28d9 2024-01-23T08:51:52.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.163+0000 7fcb212a3700 1 -- 172.21.15.59:0/1374594586 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fcb04068f60 msgr2=0x7fcb0406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:52.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.163+0000 7fcb212a3700 1 --2- 172.21.15.59:0/1374594586 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fcb04068f60 0x7fcb0406b410 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fcb10004f40 tx=0x7fcb1000b040 comp rx=0 tx=0).stop 2024-01-23T08:51:52.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.163+0000 7fcb212a3700 1 -- 172.21.15.59:0/1374594586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb1c0fd910 msgr2=0x7fcb1c10d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:52.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.163+0000 7fcb212a3700 1 --2- 172.21.15.59:0/1374594586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb1c0fd910 0x7fcb1c10d7f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c009360 tx=0x7fcb0c009e30 comp rx=0 tx=0).stop 2024-01-23T08:51:52.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.163+0000 7fcb212a3700 1 -- 172.21.15.59:0/1374594586 shutdown_connections 2024-01-23T08:51:52.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.163+0000 7fcb212a3700 1 --2- 172.21.15.59:0/1374594586 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7fcb04068f60 0x7fcb0406b410 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:52.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.163+0000 7fcb212a3700 1 --2- 172.21.15.59:0/1374594586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb1c0fd910 0x7fcb1c10d7f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:52.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.163+0000 7fcb212a3700 1 --2- 172.21.15.59:0/1374594586 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb1c100230 0x7fcb1c10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:52.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.163+0000 7fcb212a3700 1 -- 172.21.15.59:0/1374594586 >> 172.21.15.59:0/1374594586 conn(0x7fcb1c0f7370 msgr2=0x7fcb1c0f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:51:52.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.163+0000 7fcb212a3700 1 -- 172.21.15.59:0/1374594586 shutdown_connections 2024-01-23T08:51:52.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.163+0000 7fcb212a3700 1 -- 172.21.15.59:0/1374594586 wait complete. 2024-01-23T08:51:52.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.294+0000 7f1d412b2700 1 -- 172.21.15.59:0/318631246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d3c0fdc20 msgr2=0x7f1d3c0fc030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:52.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.294+0000 7f1d412b2700 1 --2- 172.21.15.59:0/318631246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d3c0fdc20 0x7f1d3c0fc030 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1d30004940 tx=0x7f1d30009e30 comp rx=0 tx=0).stop 2024-01-23T08:51:52.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.294+0000 7f1d412b2700 1 -- 172.21.15.59:0/318631246 shutdown_connections 2024-01-23T08:51:52.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.294+0000 7f1d412b2700 1 --2- 172.21.15.59:0/318631246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d3c0fdc20 0x7f1d3c0fc030 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:52.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.294+0000 7f1d412b2700 1 --2- 172.21.15.59:0/318631246 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d3c0fd2d0 0x7f1d3c0fd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:52.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.294+0000 7f1d412b2700 1 -- 172.21.15.59:0/318631246 >> 172.21.15.59:0/318631246 conn(0x7f1d3c0f6bb0 msgr2=0x7f1d3c0f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:51:52.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.294+0000 7f1d412b2700 1 -- 172.21.15.59:0/318631246 shutdown_connections 2024-01-23T08:51:52.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.294+0000 7f1d412b2700 1 -- 172.21.15.59:0/318631246 wait complete. 2024-01-23T08:51:52.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.295+0000 7f1d412b2700 1 Processor -- start 2024-01-23T08:51:52.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.295+0000 7f1d412b2700 1 -- start start 2024-01-23T08:51:52.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.295+0000 7f1d412b2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d3c0fd2d0 0x7f1d3c10d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:52.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.295+0000 7f1d412b2700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d3c0fdc20 0x7f1d3c10dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:52.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.295+0000 7f1d412b2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c10e4e0 con 0x7f1d3c0fd2d0 2024-01-23T08:51:52.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.295+0000 7f1d412b2700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c10ed80 con 0x7f1d3c0fdc20 2024-01-23T08:51:52.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.295+0000 7f1d3affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d3c0fd2d0 0x7f1d3c10d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:52.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.295+0000 7f1d3affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d3c0fd2d0 0x7f1d3c10d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60212/0 (socket says 172.21.15.59:60212) 2024-01-23T08:51:52.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.295+0000 7f1d3affd700 1 -- 172.21.15.59:0/3831162901 learned_addr learned my addr 172.21.15.59:0/3831162901 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:51:52.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.295+0000 7f1d3a7fc700 1 --2- 172.21.15.59:0/3831162901 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d3c0fdc20 0x7f1d3c10dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:52.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.295+0000 7f1d3affd700 1 -- 172.21.15.59:0/3831162901 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d3c0fdc20 msgr2=0x7f1d3c10dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:52.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.295+0000 7f1d3affd700 1 --2- 172.21.15.59:0/3831162901 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d3c0fdc20 0x7f1d3c10dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:52.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.295+0000 7f1d3affd700 1 -- 172.21.15.59:0/3831162901 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d30005040 con 0x7f1d3c0fd2d0 2024-01-23T08:51:52.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.296+0000 7f1d3affd700 1 --2- 172.21.15.59:0/3831162901 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d3c0fd2d0 0x7f1d3c10d9a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c00eb40 tx=0x7f1d2c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:51:52.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.296+0000 7f1d237fe700 1 -- 172.21.15.59:0/3831162901 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d2c007cc0 con 0x7f1d3c0fd2d0 2024-01-23T08:51:52.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.296+0000 7f1d237fe700 1 -- 172.21.15.59:0/3831162901 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d2c00f040 con 0x7f1d3c0fd2d0 2024-01-23T08:51:52.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.296+0000 7f1d412b2700 1 -- 172.21.15.59:0/3831162901 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d3c10f000 con 0x7f1d3c0fd2d0 2024-01-23T08:51:52.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.296+0000 7f1d237fe700 1 -- 172.21.15.59:0/3831162901 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d2c00c990 con 0x7f1d3c0fd2d0 2024-01-23T08:51:52.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.296+0000 7f1d412b2700 1 -- 172.21.15.59:0/3831162901 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d3c10f540 con 0x7f1d3c0fd2d0 2024-01-23T08:51:52.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.297+0000 7f1d412b2700 1 -- 172.21.15.59:0/3831162901 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d3c04bd60 con 0x7f1d3c0fd2d0 2024-01-23T08:51:52.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.297+0000 7f1d237fe700 1 -- 172.21.15.59:0/3831162901 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f1d2c00caf0 con 0x7f1d3c0fd2d0 2024-01-23T08:51:52.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.298+0000 7f1d237fe700 1 --2- 172.21.15.59:0/3831162901 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f1d24068f60 0x7f1d2406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:52.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.298+0000 7f1d237fe700 1 -- 172.21.15.59:0/3831162901 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f1d2c086900 con 0x7f1d3c0fd2d0 2024-01-23T08:51:52.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.298+0000 7f1d3a7fc700 1 --2- 172.21.15.59:0/3831162901 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f1d24068f60 0x7f1d2406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:52.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.299+0000 7f1d3a7fc700 1 --2- 172.21.15.59:0/3831162901 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f1d24068f60 0x7f1d2406b410 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1d30013680 tx=0x7f1d30009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:51:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.306+0000 7f1d237fe700 1 -- 172.21.15.59:0/3831162901 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1d2c04da00 con 0x7f1d3c0fd2d0 2024-01-23T08:51:52.310 INFO:tasks.workunit.client.1.smithi137.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-01-23T08:51:52.310 INFO:tasks.workunit.client.1.smithi137.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-01-23T08:51:52.310 INFO:tasks.workunit.client.1.smithi137.stderr:+ make 2024-01-23T08:51:52.331 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Jan 23 08:51:52 smithi137 ceph-mon[35787]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:51:52.470 INFO:tasks.workunit.client.1.smithi137.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-01-23T08:51:52.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:51:52 smithi059 ceph-mon[28424]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-23T08:51:52.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.632+0000 7f1d412b2700 1 -- 172.21.15.59:0/3831162901 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1d3c10f7b0 con 0x7f1d3c0fd2d0 2024-01-23T08:51:52.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.633+0000 7f1d237fe700 1 -- 172.21.15.59:0/3831162901 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f1d2c0568c0 con 0x7f1d3c0fd2d0 2024-01-23T08:51:52.630 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-01-23T08:51:52.630 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": { 2024-01-23T08:51:52.630 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-397-gb824e5be (b824e5befce1d427fda2bc90ae52665f0c7e2770) quincy (stable)": 2 2024-01-23T08:51:52.630 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-01-23T08:51:52.630 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": { 2024-01-23T08:51:52.630 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-397-gb824e5be (b824e5befce1d427fda2bc90ae52665f0c7e2770) quincy (stable)": 2 2024-01-23T08:51:52.630 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-01-23T08:51:52.630 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": { 2024-01-23T08:51:52.630 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-397-gb824e5be (b824e5befce1d427fda2bc90ae52665f0c7e2770) quincy (stable)": 6 2024-01-23T08:51:52.631 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-01-23T08:51:52.631 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": { 2024-01-23T08:51:52.631 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-397-gb824e5be (b824e5befce1d427fda2bc90ae52665f0c7e2770) quincy (stable)": 4 2024-01-23T08:51:52.631 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-01-23T08:51:52.631 INFO:teuthology.orchestra.run.smithi059.stdout: "overall": { 2024-01-23T08:51:52.631 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-397-gb824e5be (b824e5befce1d427fda2bc90ae52665f0c7e2770) quincy (stable)": 14 2024-01-23T08:51:52.631 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-01-23T08:51:52.631 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-01-23T08:51:52.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.635+0000 7f1d412b2700 1 -- 172.21.15.59:0/3831162901 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f1d24068f60 msgr2=0x7f1d2406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:52.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.635+0000 7f1d412b2700 1 --2- 172.21.15.59:0/3831162901 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f1d24068f60 0x7f1d2406b410 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1d30013680 tx=0x7f1d30009e30 comp rx=0 tx=0).stop 2024-01-23T08:51:52.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.635+0000 7f1d412b2700 1 -- 172.21.15.59:0/3831162901 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d3c0fd2d0 msgr2=0x7f1d3c10d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:52.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.635+0000 7f1d412b2700 1 --2- 172.21.15.59:0/3831162901 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d3c0fd2d0 0x7f1d3c10d9a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c00eb40 tx=0x7f1d2c00b040 comp rx=0 tx=0).stop 2024-01-23T08:51:52.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.635+0000 7f1d412b2700 1 -- 172.21.15.59:0/3831162901 shutdown_connections 2024-01-23T08:51:52.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.635+0000 7f1d412b2700 1 --2- 172.21.15.59:0/3831162901 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f1d24068f60 0x7f1d2406b410 secure :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1d30013680 tx=0x7f1d30009e30 comp rx=0 tx=0).stop 2024-01-23T08:51:52.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.635+0000 7f1d412b2700 1 --2- 172.21.15.59:0/3831162901 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d3c0fd2d0 0x7f1d3c10d9a0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:52.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.635+0000 7f1d412b2700 1 --2- 172.21.15.59:0/3831162901 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d3c0fdc20 0x7f1d3c10dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:52.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.635+0000 7f1d412b2700 1 -- 172.21.15.59:0/3831162901 >> 172.21.15.59:0/3831162901 conn(0x7f1d3c0f6bb0 msgr2=0x7f1d3c0fafc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:51:52.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.635+0000 7f1d412b2700 1 -- 172.21.15.59:0/3831162901 shutdown_connections 2024-01-23T08:51:52.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.635+0000 7f1d412b2700 1 -- 172.21.15.59:0/3831162901 wait complete. 2024-01-23T08:51:52.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.742+0000 7f310807c700 1 -- 172.21.15.59:0/2450582680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3100066ae0 msgr2=0x7f31000ff9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:52.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.742+0000 7f310807c700 1 --2- 172.21.15.59:0/2450582680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3100066ae0 0x7f31000ff9a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f30fc004940 tx=0x7f30fc009e30 comp rx=0 tx=0).stop 2024-01-23T08:51:52.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.742+0000 7f310807c700 1 -- 172.21.15.59:0/2450582680 shutdown_connections 2024-01-23T08:51:52.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.742+0000 7f310807c700 1 --2- 172.21.15.59:0/2450582680 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31000fff70 0x7f31001023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:52.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.742+0000 7f310807c700 1 --2- 172.21.15.59:0/2450582680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3100066ae0 0x7f31000ff9a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:52.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.742+0000 7f310807c700 1 -- 172.21.15.59:0/2450582680 >> 172.21.15.59:0/2450582680 conn(0x7f31000f7390 msgr2=0x7f31000f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:51:52.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.742+0000 7f310807c700 1 -- 172.21.15.59:0/2450582680 shutdown_connections 2024-01-23T08:51:52.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.743+0000 7f310807c700 1 -- 172.21.15.59:0/2450582680 wait complete. 2024-01-23T08:51:52.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.743+0000 7f310807c700 1 Processor -- start 2024-01-23T08:51:52.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.743+0000 7f310807c700 1 -- start start 2024-01-23T08:51:52.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.743+0000 7f310807c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3100066ae0 0x7f310010d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:52.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.743+0000 7f310807c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31000fff70 0x7f310010dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:52.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.743+0000 7f310807c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310010e360 con 0x7f31000fff70 2024-01-23T08:51:52.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.743+0000 7f310807c700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310010ec00 con 0x7f3100066ae0 2024-01-23T08:51:52.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.743+0000 7f3105617700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31000fff70 0x7f310010dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:52.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.743+0000 7f3105617700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31000fff70 0x7f310010dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60218/0 (socket says 172.21.15.59:60218) 2024-01-23T08:51:52.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.743+0000 7f3105617700 1 -- 172.21.15.59:0/2572107209 learned_addr learned my addr 172.21.15.59:0/2572107209 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-01-23T08:51:52.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.744+0000 7f3105e18700 1 --2- 172.21.15.59:0/2572107209 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3100066ae0 0x7f310010d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:52.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.744+0000 7f3105617700 1 -- 172.21.15.59:0/2572107209 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3100066ae0 msgr2=0x7f310010d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:52.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.744+0000 7f3105617700 1 --2- 172.21.15.59:0/2572107209 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3100066ae0 0x7f310010d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:52.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.744+0000 7f3105617700 1 -- 172.21.15.59:0/2572107209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30fc005040 con 0x7f31000fff70 2024-01-23T08:51:52.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.744+0000 7f3105617700 1 --2- 172.21.15.59:0/2572107209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31000fff70 0x7f310010dd60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f30f000eb40 tx=0x7f30f000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:51:52.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.744+0000 7f30f67fc700 1 -- 172.21.15.59:0/2572107209 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30f0007cc0 con 0x7f31000fff70 2024-01-23T08:51:52.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.744+0000 7f310807c700 1 -- 172.21.15.59:0/2572107209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f310010ee80 con 0x7f31000fff70 2024-01-23T08:51:52.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.744+0000 7f310807c700 1 -- 172.21.15.59:0/2572107209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f310010f3c0 con 0x7f31000fff70 2024-01-23T08:51:52.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.744+0000 7f30f67fc700 1 -- 172.21.15.59:0/2572107209 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f30f000f040 con 0x7f31000fff70 2024-01-23T08:51:52.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.744+0000 7f30f67fc700 1 -- 172.21.15.59:0/2572107209 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30f000c8f0 con 0x7f31000fff70 2024-01-23T08:51:52.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.746+0000 7f310807c700 1 -- 172.21.15.59:0/2572107209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e40052e0 con 0x7f31000fff70 2024-01-23T08:51:52.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.746+0000 7f30f67fc700 1 -- 172.21.15.59:0/2572107209 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f30f000ca50 con 0x7f31000fff70 2024-01-23T08:51:52.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.746+0000 7f30f67fc700 1 --2- 172.21.15.59:0/2572107209 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f30ec068f60 0x7f30ec06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-23T08:51:52.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.746+0000 7f30f67fc700 1 -- 172.21.15.59:0/2572107209 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f30f0086680 con 0x7f31000fff70 2024-01-23T08:51:52.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.747+0000 7f3105e18700 1 --2- 172.21.15.59:0/2572107209 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f30ec068f60 0x7f30ec06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-23T08:51:52.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.747+0000 7f3105e18700 1 --2- 172.21.15.59:0/2572107209 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f30ec068f60 0x7f30ec06b410 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f30fc000c00 tx=0x7f30fc009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-23T08:51:52.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.751+0000 7f30f67fc700 1 -- 172.21.15.59:0/2572107209 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f30f004da00 con 0x7f31000fff70 2024-01-23T08:51:52.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.981+0000 7f310807c700 1 -- 172.21.15.59:0/2572107209 --> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f30e4000bf0 con 0x7f30ec068f60 2024-01-23T08:51:52.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.982+0000 7f30f67fc700 1 -- 172.21.15.59:0/2572107209 <== mgr.14210 v2:172.21.15.59:6800/3219735162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7f30e4000bf0 con 0x7f30ec068f60 2024-01-23T08:51:52.979 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-01-23T08:51:52.979 INFO:teuthology.orchestra.run.smithi059.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7", 2024-01-23T08:51:52.979 INFO:teuthology.orchestra.run.smithi059.stdout: "in_progress": true, 2024-01-23T08:51:52.979 INFO:teuthology.orchestra.run.smithi059.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-01-23T08:51:52.979 INFO:teuthology.orchestra.run.smithi059.stdout: "services_complete": [], 2024-01-23T08:51:52.979 INFO:teuthology.orchestra.run.smithi059.stdout: "progress": "", 2024-01-23T08:51:52.979 INFO:teuthology.orchestra.run.smithi059.stdout: "message": "Currently upgrading mgr daemons", 2024-01-23T08:51:52.979 INFO:teuthology.orchestra.run.smithi059.stdout: "is_paused": false 2024-01-23T08:51:52.979 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-01-23T08:51:52.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.984+0000 7f310807c700 1 -- 172.21.15.59:0/2572107209 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f30ec068f60 msgr2=0x7f30ec06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:52.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.984+0000 7f310807c700 1 --2- 172.21.15.59:0/2572107209 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f30ec068f60 0x7f30ec06b410 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f30fc000c00 tx=0x7f30fc009e30 comp rx=0 tx=0).stop 2024-01-23T08:51:52.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.984+0000 7f310807c700 1 -- 172.21.15.59:0/2572107209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31000fff70 msgr2=0x7f310010dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-23T08:51:52.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.984+0000 7f310807c700 1 --2- 172.21.15.59:0/2572107209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31000fff70 0x7f310010dd60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f30f000eb40 tx=0x7f30f000b040 comp rx=0 tx=0).stop 2024-01-23T08:51:52.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.984+0000 7f310807c700 1 -- 172.21.15.59:0/2572107209 shutdown_connections 2024-01-23T08:51:52.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.984+0000 7f310807c700 1 --2- 172.21.15.59:0/2572107209 >> [v2:172.21.15.59:6800/3219735162,v1:172.21.15.59:6801/3219735162] conn(0x7f30ec068f60 0x7f30ec06b410 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:52.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.984+0000 7f310807c700 1 --2- 172.21.15.59:0/2572107209 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3100066ae0 0x7f310010d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:52.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.984+0000 7f310807c700 1 --2- 172.21.15.59:0/2572107209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31000fff70 0x7f310010dd60 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-23T08:51:52.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.984+0000 7f310807c700 1 -- 172.21.15.59:0/2572107209 >> 172.21.15.59:0/2572107209 conn(0x7f31000f7390 msgr2=0x7f31000f97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-23T08:51:52.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.984+0000 7f310807c700 1 -- 172.21.15.59:0/2572107209 shutdown_connections 2024-01-23T08:51:52.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:51:52.984+0000 7f310807c700 1 -- 172.21.15.59:0/2572107209 wait complete. 2024-01-23T08:52:13.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:13.731+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:13.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:13.731+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:13.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:13.731+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:14.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:14.040+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:14.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:14.040+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:14.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:14.040+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:14.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:14.682+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:14.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:14.682+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:14.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:14.682+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:15.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:15.016+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:15.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:15.016+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:15.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:15.016+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:15.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:15.669+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:15.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:15.669+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:15.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:15.669+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:16.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:16.005+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:16.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:16.005+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:16.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:16.005+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:16.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:15.989+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:16.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:15.989+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:16.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:15.989+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:16.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:16.665+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:16.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:16.665+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:16.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:16.665+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:17.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:16.948+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:17.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:16.948+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:17.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:16.948+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:17.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:16.986+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:17.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:16.986+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:17.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:16.986+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:17.964 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:17.680+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:17.964 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:17.681+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:17.964 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:17.681+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:18.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:17.993+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:18.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:17.993+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:18.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:17.993+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:18.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:17.966+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:18.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:17.966+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:18.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:17.966+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:18.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:18.916+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:18.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:18.916+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:18.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:18.916+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:18.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:18.705+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:18.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:18.705+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:18.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:18.705+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:19.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:19.007+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:19.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:19.007+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:19.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:19.007+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:20.074 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:19.740+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:20.074 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:19.740+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:20.075 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:19.740+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:20.075 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:19.971+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:20.075 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:19.971+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:20.075 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:19.971+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:20.075 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:19.950+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:20.075 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:19.950+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:20.075 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:19.950+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:20.962 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:20.720+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:20.962 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:20.720+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:20.962 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:20.720+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:21.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:20.994+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:21.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:20.994+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:21.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:20.994+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:21.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:20.964+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:21.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:20.964+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:21.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:20.964+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:22.010 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:21.976+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:22.010 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:21.976+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:22.010 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:21.976+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:22.010 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:21.738+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:22.010 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:21.738+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:22.010 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:21.738+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:22.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:22.013+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:22.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:22.013+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:22.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:22.013+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:23.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:22.751+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:23.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:22.751+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:23.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:22.751+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:23.022 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:22.959+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:23.022 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:22.959+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:23.022 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:22.959+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:23.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:23.024+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:23.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:23.024+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:23.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:23.024+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:23.937 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:23.712+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:23.937 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:23.712+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:23.937 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:23.712+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:24.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:23.940+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:24.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:23.940+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:24.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:23.940+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:24.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:24.063+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:24.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:24.063+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:24.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:24.063+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:24.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:24.960+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:24.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:24.960+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:24.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:24.960+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:24.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:24.717+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:24.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:24.717+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:24.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:24.717+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:25.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:25.068+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:25.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:25.068+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:25.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:25.068+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:26.100 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:25.726+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:26.100 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:25.726+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:26.100 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:25.726+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:26.100 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:25.932+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:26.100 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:25.932+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:26.100 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:25.932+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:26.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:26.102+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:26.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:26.102+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:26.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:26.102+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:26.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:26.933+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:26.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:26.933+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:26.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:26.933+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:26.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:26.708+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:26.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:26.708+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:26.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:26.708+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:27.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:27.096+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:27.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:27.096+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:27.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:27.096+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:27.918 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:27.663+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:27.918 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:27.663+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:27.918 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:27.663+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:28.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:27.921+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:28.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:27.921+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:28.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:27.921+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:28.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:28.107+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:28.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:28.107+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:28.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:28.107+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:28.876 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:28.624+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:28.876 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:28.624+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:28.876 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:28.624+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:29.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:28.879+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:29.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:28.879+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:29.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:28.879+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:29.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:29.100+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:29.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:29.100+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:29.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:29.100+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:29.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:29.628+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:29.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:29.628+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:29.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:29.628+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:29.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:29.851+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:29.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:29.851+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:29.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:29.851+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:30.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:30.079+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:30.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:30.079+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:30.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:30.079+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:30.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:30.872+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:30.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:30.872+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:30.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:30.872+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:30.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:30.671+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:30.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:30.671+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:30.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:30.671+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:31.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:31.060+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:31.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:31.060+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:31.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:31.060+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:31.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:31.855+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:31.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:31.855+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:31.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:31.855+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:31.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:31.671+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:31.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:31.671+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:31.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:31.671+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:32.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:32.036+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:32.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:32.036+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:32.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:32.036+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:32.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:32.806+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:32.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:32.806+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:32.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:32.806+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:32.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:32.624+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:32.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:32.624+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:32.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:32.624+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:33.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:33.059+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:33.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:33.059+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:33.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:33.059+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:33.842 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:33.582+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:33.843 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:33.582+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:33.843 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:33.582+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:34.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:34.043+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:34.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:34.043+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:34.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:34.043+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:34.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:33.845+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:34.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:33.845+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:34.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:33.845+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:34.889 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:34.542+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:34.889 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:34.542+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:34.889 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:34.542+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:35.190 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:34.892+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:35.190 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:34.892+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:35.190 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:34.892+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:35.190 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:35.044+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:35.190 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:35.044+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:35.190 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:35.044+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:35.903 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:35.540+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:35.903 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:35.540+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:35.903 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:35.540+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:36.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:35.906+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:36.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:35.906+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:36.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:35.906+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:36.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:35.999+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:36.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:35.999+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:36.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:35.999+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:36.925 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:36.583+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:36.925 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:36.583+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:36.925 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:36.583+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:37.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:36.927+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:37.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:36.927+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:37.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:36.927+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:37.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:37.015+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:37.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:37.015+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:37.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:37.015+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:37.898 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:37.619+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:37.898 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:37.619+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:37.898 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:37.619+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:38.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:38.057+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:38.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:38.057+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:38.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:38.057+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:38.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:37.900+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:38.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:37.900+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:38.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:37.900+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:38.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:38.638+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:38.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:38.638+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:38.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:38.638+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:38.872+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:38.872+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:38.872+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:39.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:39.038+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:39.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:39.038+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:39.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:39.038+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:39.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:39.660+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:39.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:39.660+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:39.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:39.660+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:39.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:39.895+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:39.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:39.895+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:39.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:39.896+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:40.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:40.036+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:40.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:40.036+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:40.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:40.036+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:40.912 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:40.639+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:40.913 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:40.639+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:40.913 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:40.639+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:41.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:40.915+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:41.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:40.915+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:41.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:40.915+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:41.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:41.048+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:41.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:41.048+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:41.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:41.048+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:41.923 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:41.634+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:41.923 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:41.634+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:41.923 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:41.634+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:42.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:42.062+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:42.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:42.062+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:42.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:42.062+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:42.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:41.926+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:42.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:41.926+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:42.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:41.926+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:42.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:42.654+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:42.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:42.654+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:42.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:42.654+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:43.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:43.099+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:43.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:43.099+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:43.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:43.099+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:43.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:42.947+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:43.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:42.947+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:43.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:42.947+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:43.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:43.617+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:43.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:43.617+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:43.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:43.617+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:44.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:43.975+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:44.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:43.975+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:44.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:43.975+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:44.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:44.107+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:44.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:44.107+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:44.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:44.107+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:44.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:44.650+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:44.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:44.650+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:44.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:44.650+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:45.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:45.099+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:45.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:45.099+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:45.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:45.099+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:45.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:44.988+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:45.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:44.988+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:45.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:44.988+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:45.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:45.688+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:45.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:45.688+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:45.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:45.688+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:46.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:45.987+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:46.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:45.988+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:46.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:45.988+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:46.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:46.072+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:46.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:46.072+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:46.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:46.072+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:47.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:46.986+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:47.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:46.986+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:47.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:46.986+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:47.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:46.737+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:47.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:46.737+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:47.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:46.737+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:47.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:47.091+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:47.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:47.091+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:47.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:47.091+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:47.992 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:47.728+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:47.992 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:47.728+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:47.992 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:47.728+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:48.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:47.994+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:48.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:47.994+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:48.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:47.994+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:48.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:48.131+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:48.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:48.131+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:48.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:48.131+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:48.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:48.694+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:48.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:48.694+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:48.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:48.694+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:49.243 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:48.986+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:49.243 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:48.986+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:49.243 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:48.986+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:49.244 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:49.165+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:49.244 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:49.165+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:49.244 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:49.165+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:50.077 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:49.727+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:50.078 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:49.727+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:50.078 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:49.727+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:50.078 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:49.953+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:50.078 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:49.953+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:50.078 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:49.953+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:50.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:50.197+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:50.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:50.197+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:50.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:50.197+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:51.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:50.769+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:51.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:50.769+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:51.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:50.769+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:51.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:50.999+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:51.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:50.999+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:51.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:50.999+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:51.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:51.239+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:51.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:51.239+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:51.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:51.239+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:52.041 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:51.774+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:52.042 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:51.774+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:52.042 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:51.774+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:52.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:52.044+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:52.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:52.044+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:52.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:52.044+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:52.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:52.205+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:52.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:52.205+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:52.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:52.205+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:53.086 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:52.746+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:53.086 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:52.746+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:53.086 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:52.746+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:53.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:53.183+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:53.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:53.183+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:53.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:53.183+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:53.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:53.088+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:53.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:53.088+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:53.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:53.088+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:53.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:53.759+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:53.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:53.759+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:53.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:53.759+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:54.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:54.186+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:54.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:54.186+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:54.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:54.186+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:54.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:54.083+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:54.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:54.083+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:54.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:54.083+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:55.077 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:54.765+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:55.077 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:54.765+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:55.077 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:54.765+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:55.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:55.194+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:55.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:55.194+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:55.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:55.194+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:55.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:55.097+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:55.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:55.097+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:55.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:55.097+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:56.095 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:55.783+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:56.096 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:55.783+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:56.096 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:55.783+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:56.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:56.213+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:56.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:56.213+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:56.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:56.213+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:56.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:56.096+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:56.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:56.096+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:56.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:56.096+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:57.102 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:56.782+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:57.102 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:56.782+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:57.102 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:56.782+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:57.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:57.213+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:57.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:57.213+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:57.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:57.213+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:57.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:57.105+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:57.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:57.105+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:57.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:57.105+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:58.098 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:57.782+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:58.098 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:57.782+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:58.098 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:57.782+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:58.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:58.101+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:58.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:58.101+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:58.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:58.101+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:58.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:58.202+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:58.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:58.202+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:58.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:58.202+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:59.065 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:58.813+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:59.065 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:58.813+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:59.065 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:58.813+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:52:59.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:59.184+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:59.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:59.184+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:59.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:52:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:52:59.184+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:52:59.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:59.068+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:59.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:59.068+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:52:59.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:52:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:52:59.068+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:00.078 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:59.805+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:00.078 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:59.805+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:00.078 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:52:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:52:59.805+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:00.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:00.101+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:00.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:00.101+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:00.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:00.101+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:00.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:00.217+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:00.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:00.217+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:00.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:00.217+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:01.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:00.819+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:01.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:00.819+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:01.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:00.819+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:01.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:01.061+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:01.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:01.061+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:01.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:01.061+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:01.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:01.204+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:01.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:01.204+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:01.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:01.204+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:02.044 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:01.779+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:02.045 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:01.779+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:02.045 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:01.779+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:02.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:02.168+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:02.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:02.168+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:02.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:02.168+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:02.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:02.047+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:02.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:02.047+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:02.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:02.047+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:03.168 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:03.010+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:03.168 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:03.010+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:03.168 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:03.010+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:03.169 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:02.826+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:03.169 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:02.826+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:03.169 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:02.826+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:03.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:03.171+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:03.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:03.171+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:03.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:03.171+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:04.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:03.782+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:04.171 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:03.782+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:04.171 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:03.782+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:04.171 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:04.014+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:04.171 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:04.014+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:04.171 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:04.014+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:04.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:04.173+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:04.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:04.173+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:04.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:04.173+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:05.080 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:04.799+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:05.080 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:04.799+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:05.080 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:04.799+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:05.080 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:04.988+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:05.080 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:04.989+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:05.080 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:04.989+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:05.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:05.179+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:05.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:05.179+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:05.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:05.179+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:06.153 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:06.007+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:06.153 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:06.007+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:06.153 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:06.008+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:06.153 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:05.810+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:06.153 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:05.810+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:06.153 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:05.810+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:06.435 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:06.155+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:06.435 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:06.155+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:06.435 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:06.155+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:07.112 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:06.812+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:07.112 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:06.812+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:07.112 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:06.812+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:07.113 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:07.008+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:07.113 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:07.008+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:07.113 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:07.008+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:07.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:07.115+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:07.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:07.115+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:07.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:07.115+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:08.093 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:08.025+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:08.094 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:08.025+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:08.094 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:08.025+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:08.094 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:07.827+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:08.094 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:07.827+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:08.094 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:07.827+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:08.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:08.096+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:08.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:08.096+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:08.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:08.096+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:09.139 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:09.013+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:09.139 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:09.013+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:09.139 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:09.013+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:09.139 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:08.850+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:09.139 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:08.850+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:09.140 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:08.850+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:09.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:09.142+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:09.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:09.142+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:09.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:09.142+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:10.191 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:10.004+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:10.191 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:10.004+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:10.191 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:10.004+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:10.191 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:09.899+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:10.191 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:09.899+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:10.191 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:09.899+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:10.192 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:10.131+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:10.192 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:10.131+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:10.192 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:10.131+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:11.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:10.932+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:11.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:10.932+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:11.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:10.932+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:11.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:11.122+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:11.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:11.122+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:11.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:11.122+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:11.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:11.026+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:11.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:11.026+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:11.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:11.026+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:12.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:11.897+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:12.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:11.897+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:12.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:11.897+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:12.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:12.037+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:12.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:12.037+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:12.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:12.037+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:12.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:12.146+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:12.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:12.146+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:12.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:12.146+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:13.168 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:12.991+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:13.168 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:12.991+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:13.168 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:12.991+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:13.168 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:12.848+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:13.168 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:12.848+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:13.168 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:12.848+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:13.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:13.171+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:13.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:13.171+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:13.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:13.171+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:14.179 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:13.961+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:14.179 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:13.961+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:14.179 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:13.961+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:14.179 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:13.862+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:14.179 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:13.862+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:14.179 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:13.862+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:14.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:14.182+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:14.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:14.182+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:14.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:14.182+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:14.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:14.817+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:14.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:14.817+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:14.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:14.817+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:15.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:14.938+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:15.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:14.938+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:15.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:14.938+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:15.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:15.222+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:15.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:15.222+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:15.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:15.222+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:15.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:15.799+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:15.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:15.799+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:15.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:15.799+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:16.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:15.949+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:16.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:15.949+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:16.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:15.949+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:16.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:16.238+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:16.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:16.238+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:16.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:16.238+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:17.203 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:16.785+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:17.203 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:16.785+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:17.203 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:16.785+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:17.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:16.918+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:17.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:16.918+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:17.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:16.918+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:17.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:17.205+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:17.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:17.205+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:17.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:17.205+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:18.211 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:17.756+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:18.211 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:17.756+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:18.211 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:17.756+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:18.211 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:17.968+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:18.211 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:17.968+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:18.212 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:17.968+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:18.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:18.213+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:18.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:18.213+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:18.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:18.213+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:19.008 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:18.733+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:19.009 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:18.733+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:19.009 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:18.733+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:19.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:19.011+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:19.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:19.011+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:19.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:19.011+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:19.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:19.203+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:19.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:19.203+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:19.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:19.203+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:19.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:19.700+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:19.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:19.700+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:19.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:19.700+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:20.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:20.029+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:20.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:20.029+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:20.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:20.029+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:20.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:20.238+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:20.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:20.238+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:20.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:20.238+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:21.009 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:20.727+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:21.009 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:20.727+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:21.009 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:20.727+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:21.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:21.235+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:21.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:21.235+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:21.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:21.235+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:21.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:21.010+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:21.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:21.010+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:21.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:21.010+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:22.217 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:21.774+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:22.217 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:21.774+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:22.218 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:21.774+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:22.218 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:21.997+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:22.218 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:21.997+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:22.218 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:21.997+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:22.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:22.219+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:22.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:22.219+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:22.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:22.219+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:23.044 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:22.794+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:23.044 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:22.794+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:23.044 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:22.794+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:23.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:23.259+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:23.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:23.259+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:23.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:23.259+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:23.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:23.046+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:23.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:23.046+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:23.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:23.046+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:24.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:23.777+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:24.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:23.777+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:24.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:23.777+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:24.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:24.002+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:24.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:24.002+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:24.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:24.002+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:24.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:24.272+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:24.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:24.272+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:24.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:24.272+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:25.080 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:24.784+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:25.080 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:24.784+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:25.080 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:24.784+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:25.080 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:24.972+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:25.081 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:24.972+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:25.081 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:24.973+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:25.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:25.281+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:25.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:25.281+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:25.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:25.281+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:26.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:25.813+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:26.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:25.813+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:26.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:25.813+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:26.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:25.939+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:26.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:25.939+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:26.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:25.939+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:26.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:26.235+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:26.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:26.235+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:26.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:26.235+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:27.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:26.798+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:27.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:26.798+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:27.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:26.798+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:27.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:26.927+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:27.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:26.927+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:27.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:26.927+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:27.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:27.253+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:27.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:27.253+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:27.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:27.253+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:28.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:27.813+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:28.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:27.813+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:28.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:27.813+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:28.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:27.901+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:28.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:27.901+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:28.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:27.901+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:28.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:28.268+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:28.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:28.268+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:28.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:28.268+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:29.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:28.798+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:29.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:28.798+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:29.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:28.798+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:29.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:28.855+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:29.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:28.855+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:29.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:28.855+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:29.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:29.269+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:29.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:29.269+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:29.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:29.269+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:30.072 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:29.894+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:30.072 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:29.894+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:30.072 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:29.894+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:30.072 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:29.786+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:30.072 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:29.786+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:30.072 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:29.786+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:30.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:30.243+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:30.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:30.243+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:30.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:30.243+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:31.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:30.922+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:31.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:30.922+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:31.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:30.922+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:31.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:30.795+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:31.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:30.795+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:31.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:30.795+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:31.621 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:31.292+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:31.621 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:31.292+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:31.621 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:31.292+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:31.891 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:31.781+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:31.891 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:31.781+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:31.891 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:31.781+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:32.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:31.893+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:32.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:31.893+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:32.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:31.893+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:32.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:32.292+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:32.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:32.292+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:32.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:32.292+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:33.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:32.857+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:33.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:32.857+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:33.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:32.857+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:33.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:32.799+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:33.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:32.799+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:33.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:32.799+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:33.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:33.331+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:33.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:33.331+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:33.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:33.331+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:34.083 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:33.865+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:34.083 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:33.865+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:34.083 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:33.865+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:34.083 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:33.797+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:34.083 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:33.797+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:34.083 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:33.797+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:34.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:34.378+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:34.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:34.378+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:34.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:34.378+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:35.084 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:34.756+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:35.084 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:34.756+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:35.084 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:34.756+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:35.084 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:34.817+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:35.085 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:34.817+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:35.085 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:34.817+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:35.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:35.343+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:35.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:35.343+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:35.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:35.343+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:36.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:35.759+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:36.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:35.759+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:36.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:35.759+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:36.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:35.853+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:36.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:35.853+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:36.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:35.853+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:36.715 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:36.331+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:36.715 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:36.331+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:36.716 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:36.331+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:36.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:36.717+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:36.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:36.717+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:36.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:36.717+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:36.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:36.819+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:36.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:36.819+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:36.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:36.819+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:37.684 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:37.293+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:37.684 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:37.293+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:37.684 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:37.293+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:37.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:37.687+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:37.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:37.687+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:37.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:37.687+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:37.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:37.787+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:37.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:37.787+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:37.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:37.787+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:38.686 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:38.309+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:38.686 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:38.309+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:38.686 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:38.309+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:38.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:38.688+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:38.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:38.688+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:38.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:38.688+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:38.779+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:38.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:38.779+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:38.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:38.779+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:39.688 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:39.321+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:39.688 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:39.321+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:39.688 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:39.321+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:39.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:39.691+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:39.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:39.691+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:39.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:39.691+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:39.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:39.740+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:39.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:39.740+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:39.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:39.740+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:40.649 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:40.355+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:40.649 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:40.355+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:40.649 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:40.355+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:40.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:40.775+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:40.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:40.775+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:40.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:40.775+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:40.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:40.652+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:40.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:40.652+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:40.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:40.652+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:41.621 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:41.365+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:41.622 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:41.365+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:41.622 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:41.366+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:41.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:41.690+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:41.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:41.690+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:41.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:41.690+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:41.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:41.745+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:41.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:41.745+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:41.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:41.745+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:42.693 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:42.326+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:42.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:42.326+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:42.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:42.326+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:42.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:42.716+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:42.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:42.716+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:42.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:42.716+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:42.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:42.696+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:42.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:42.696+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:42.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:42.696+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:43.675 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:43.322+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:43.675 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:43.322+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:43.675 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:43.322+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:43.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:43.678+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:43.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:43.678+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:43.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:43.678+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:43.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:43.725+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:43.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:43.725+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:43.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:43.725+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:44.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:44.358+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:44.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:44.358+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:44.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:44.358+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:44.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:44.703+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:44.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:44.703+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:44.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:44.703+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:44.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:44.713+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:44.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:44.713+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:44.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:44.713+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:45.677 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:45.407+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:45.677 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:45.407+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:45.677 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:45.407+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:45.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:45.680+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:45.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:45.680+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:45.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:45.680+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:45.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:45.759+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:45.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:45.759+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:45.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:45.759+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:46.647 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:46.408+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:46.647 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:46.408+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:46.647 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:46.408+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:46.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:46.650+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:46.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:46.650+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:46.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:46.650+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:46.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:46.710+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:46.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:46.710+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:46.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:46.710+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:47.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:47.630+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:47.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:47.631+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:47.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:47.631+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:47.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:47.425+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:47.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:47.425+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:47.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:47.425+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:48.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:47.730+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:48.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:47.730+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:48.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:47.730+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:48.672 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:48.390+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:48.672 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:48.390+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:48.672 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:48.390+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:48.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:48.675+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:48.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:48.675+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:48.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:48.675+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:48.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:48.757+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:48.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:48.757+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:48.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:48.757+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:49.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:49.412+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:49.683 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:49.412+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:49.683 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:49.412+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:49.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:49.722+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:49.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:49.722+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:49.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:49.722+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:49.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:49.685+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:49.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:49.685+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:49.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:49.685+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:50.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:50.448+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:50.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:50.448+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:50.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:50.448+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:50.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:50.704+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:50.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:50.704+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:50.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:50.704+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:50.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:50.759+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:50.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:50.759+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:50.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:50.759+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:51.709 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:51.437+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:51.709 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:51.437+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:51.709 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:51.437+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:51.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:51.769+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:51.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:51.769+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:51.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:51.769+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:51.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:51.711+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:51.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:51.711+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:51.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:51.711+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:52.692 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:52.393+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:52.693 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:52.393+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:52.693 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:52.393+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:52.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:52.801+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:52.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:52.801+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:52.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:52.801+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:52.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:52.696+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:52.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:52.696+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:52.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:52.696+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:53.676 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:53.419+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:53.676 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:53.419+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:53.677 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:53.419+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:53.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:53.847+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:53.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:53.847+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:53.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:53.847+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:53.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:53.679+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:53.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:53.679+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:53.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:53.679+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:54.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:54.407+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:54.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:54.407+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:54.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:54.407+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:54.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:54.634+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:54.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:54.634+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:54.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:54.634+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:55.074 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:54.819+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:55.074 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:54.819+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:55.074 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:54.819+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:55.443 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:53:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:53:55.077+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:53:55.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:55.621+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:55.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:55.621+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:55.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:55.621+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:55.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:55.445+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:55.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:55.446+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:55.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:55.446+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:56.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:55.841+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:56.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:55.841+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:56.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:55.841+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:56.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:56.424+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:56.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:56.424+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:56.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:56.424+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:56.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:56.650+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:56.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:56.650+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:56.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:56.650+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:57.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:56.834+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:57.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:56.834+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:57.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:56.834+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:57.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:57.471+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:57.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:57.471+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:57.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:57.471+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:57.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:57.637+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:57.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:57.637+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:57.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:57.637+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:58.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:57.808+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:58.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:57.808+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:58.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:57.808+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:58.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:58.592+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:58.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:58.592+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:58.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:58.592+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:58.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:58.446+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:58.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:58.446+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:58.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:58.446+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:59.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:58.780+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:59.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:58.780+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:59.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:58.780+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:59.551 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:59.486+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:59.551 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:59.486+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:59.551 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:53:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:53:59.486+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:53:59.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:59.554+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:59.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:59.554+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:59.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:53:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:53:59.554+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:53:59.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:59.791+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:59.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:59.791+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:53:59.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:53:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:53:59.791+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:00.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:54:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:54:00.077+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:54:00.765 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:00.580+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:00.765 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:00.580+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:00.765 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:00.580+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:00.765 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:00.487+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:00.765 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:00.487+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:00.765 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:00.487+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:01.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:00.767+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:01.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:00.767+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:01.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:00.767+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:01.731 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:01.483+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:01.731 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:01.483+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:01.731 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:01.483+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:01.731 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:01.553+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:01.732 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:01.553+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:01.732 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:01.553+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:02.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:01.736+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:02.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:01.736+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:02.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:01.736+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:02.748 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:02.592+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:02.748 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:02.592+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:02.748 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:02.592+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:02.749 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:02.477+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:02.749 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:02.477+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:02.749 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:02.477+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:03.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:02.751+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:03.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:02.751+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:03.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:02.751+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:03.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:03.572+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:03.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:03.572+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:03.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:03.572+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:03.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:03.461+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:03.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:03.461+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:03.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:03.461+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:04.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:03.736+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:04.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:03.736+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:04.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:03.736+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:04.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:04.618+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:04.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:04.618+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:04.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:04.618+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:04.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:04.441+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:04.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:04.441+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:04.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:04.441+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:05.075 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:04.736+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:05.075 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:04.736+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:05.075 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:04.736+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:05.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:54:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:54:05.078+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:54:05.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:05.572+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:05.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:05.572+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:05.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:05.572+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:05.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:05.708+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:05.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:05.708+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:05.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:05.708+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:05.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:05.484+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:05.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:05.484+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:05.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:05.484+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:06.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:06.610+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:06.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:06.610+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:06.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:06.610+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:06.436+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:06.436+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:06.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:06.436+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:07.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:06.751+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:07.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:06.751+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:07.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:06.751+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:07.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:07.658+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:07.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:07.658+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:07.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:07.658+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:07.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:07.452+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:07.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:07.453+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:07.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:07.453+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:08.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:07.744+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:08.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:07.744+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:08.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:07.744+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:08.788 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:08.621+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:08.789 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:08.621+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:08.789 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:08.621+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:08.789 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:08.478+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:08.789 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:08.478+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:08.789 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:08.478+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:09.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:08.791+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:09.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:08.791+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:09.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:08.791+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:09.757 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:09.612+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:09.789 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:09.612+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:09.789 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:09.612+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:09.789 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:09.500+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:09.789 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:09.500+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:09.789 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:09.500+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:10.075 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:09.760+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:10.076 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:09.760+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:10.076 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:09.760+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:10.466 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:54:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:54:10.078+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:54:10.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:10.597+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:10.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:10.597+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:10.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:10.597+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:10.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:10.469+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:10.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:10.469+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:10.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:10.469+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:11.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:10.732+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:11.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:10.732+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:11.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:10.732+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:11.685 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:11.646+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:11.686 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:11.646+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:11.686 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:11.646+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:11.686 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:11.424+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:11.686 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:11.424+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:11.686 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:11.424+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:11.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:11.688+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:11.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:11.688+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:11.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:11.688+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:12.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:12.409+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:12.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:12.409+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:12.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:12.409+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:12.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:12.600+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:12.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:12.600+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:12.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:12.600+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:12.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:12.654+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:12.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:12.654+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:12.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:12.654+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:13.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:13.662+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:13.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:13.662+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:13.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:13.662+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:13.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:13.598+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:13.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:13.598+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:13.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:13.598+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:13.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:13.438+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:13.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:13.438+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:13.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:13.438+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:14.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:14.479+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:14.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:14.479+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:14.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:14.479+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:14.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:14.639+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:14.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:14.639+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:14.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:14.639+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:14.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:14.614+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:14.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:14.614+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:14.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:14.614+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:15.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:54:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:54:15.079+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 57 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:54:15.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:15.631+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:15.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:15.631+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:15.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:15.631+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:15.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:15.641+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:15.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:15.641+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:15.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:15.641+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:15.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:15.526+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:15.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:15.526+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:15.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:15.526+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:16.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:16.523+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:16.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:16.523+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:16.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:16.523+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:16.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:16.678+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:16.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:16.678+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:16.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:16.678+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:16.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:16.638+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:16.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:16.638+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:16.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:16.638+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:17.872 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:17.561+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:17.872 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:17.561+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:17.872 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:17.561+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:17.872 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:17.631+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:17.872 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:17.631+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:17.873 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:17.631+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:17.873 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:17.664+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:17.873 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:17.664+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:17.873 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:17.664+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:18.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:18.615+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:18.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:18.615+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:18.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:18.615+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:18.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:18.624+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:18.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:18.624+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:18.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:18.624+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:18.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:18.533+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:18.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:18.533+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:18.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:18.533+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:19.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:19.576+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:19.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:19.576+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:19.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:19.576+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:19.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:19.579+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:19.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:19.579+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:19.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:19.579+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:19.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:19.562+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:19.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:19.562+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:19.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:19.562+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:20.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:54:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:54:20.079+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 83 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:54:20.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:20.585+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:20.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:20.585+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:20.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:20.585+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:20.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:20.549+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:20.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:20.549+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:20.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:20.549+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:20.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:20.576+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:20.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:20.576+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:20.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:20.576+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:21.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:21.547+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:21.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:21.547+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:21.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:21.547+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:21.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:21.569+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:21.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:21.569+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:21.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:21.569+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:21.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:21.623+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:21.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:21.623+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:21.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:21.623+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:22.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:22.545+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:22.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:22.545+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:22.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:22.545+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:22.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:22.578+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:22.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:22.578+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:22.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:22.578+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:22.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:22.635+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:22.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:22.635+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:22.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:22.635+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:23.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:23.553+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:23.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:23.553+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:23.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:23.553+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:23.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:23.512+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:23.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:23.512+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:23.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:23.512+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:23.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:23.665+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:23.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:23.665+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:23.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:23.665+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:24.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:24.581+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:24.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:24.581+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:24.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:24.581+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:24.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:24.512+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:24.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:24.512+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:24.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:24.512+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:24.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:24.678+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:24.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:24.678+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:24.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:24.678+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:25.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:54:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:54:25.079+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 118 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:54:25.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:25.539+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:25.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:25.539+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:25.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:25.539+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:25.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:25.715+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:25.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:25.715+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:25.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:25.715+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:25.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:25.483+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:25.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:25.483+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:25.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:25.483+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:26.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:26.512+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:26.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:26.512+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:26.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:26.512+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:26.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:26.536+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:26.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:26.536+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:26.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:26.536+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:26.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:26.701+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:26.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:26.701+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:26.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:26.701+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:27.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:27.507+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:27.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:27.507+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:27.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:27.507+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:27.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:27.554+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:27.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:27.554+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:27.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:27.554+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:27.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:27.684+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:27.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:27.684+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:27.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:27.684+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:28.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:28.658+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:28.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:28.658+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:28.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:28.658+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:28.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:28.559+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:28.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:28.559+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:28.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:28.559+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:28.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:28.483+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:28.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:28.483+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:28.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:28.483+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:29.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:29.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:29.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:29.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:29.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:29.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:29.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:29.535+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:29.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:29.535+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:29.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:29.535+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:29.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:29.505+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:29.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:29.505+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:29.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:29.505+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:30.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:54:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:54:30.080+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 145 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:54:30.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:30.582+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:30.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:30.582+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:30.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:30.582+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:30.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:30.658+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:30.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:30.658+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:30.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:30.658+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:30.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:30.535+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:30.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:30.535+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:30.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:30.535+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:31.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:31.539+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:31.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:31.539+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:31.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:31.539+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:31.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:31.571+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:31.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:31.571+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:31.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:31.571+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:31.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:31.699+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:31.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:31.699+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:31.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:31.699+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:32.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:32.578+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:32.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:32.578+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:32.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:32.578+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:32.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:32.692+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:32.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:32.692+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:32.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:32.692+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:32.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:32.607+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:32.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:32.607+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:32.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:32.607+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:33.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:33.603+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:33.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:33.603+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:33.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:33.603+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:33.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:33.740+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:33.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:33.740+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:33.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:33.740+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:33.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:33.590+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:33.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:33.590+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:33.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:33.590+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:34.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:34.580+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:34.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:34.580+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:34.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:34.580+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:34.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:34.602+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:34.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:34.602+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:34.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:34.602+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:34.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:34.770+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:34.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:34.770+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:34.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:34.770+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:35.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:54:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:54:35.080+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 175 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:54:35.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:35.583+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:35.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:35.583+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:35.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:35.583+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:35.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:35.643+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:35.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:35.643+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:35.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:35.643+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:35.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:35.725+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:35.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:35.725+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:35.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:35.725+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:36.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:36.632+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:36.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:36.632+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:36.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:36.632+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:36.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:36.599+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:36.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:36.599+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:36.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:36.599+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:36.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:36.687+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:36.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:36.687+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:36.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:36.687+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:37.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:37.609+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:37.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:37.609+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:37.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:37.609+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:37.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:37.629+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:37.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:37.629+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:37.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:37.629+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:37.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:37.645+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:37.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:37.645+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:37.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:37.645+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:38.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:38.601+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:38.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:38.601+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:38.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:38.601+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:38.583+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:38.583+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:38.583+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:38.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:38.655+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:38.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:38.655+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:38.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:38.655+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:39.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:39.550+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:39.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:39.550+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:39.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:39.550+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:39.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:39.588+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:39.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:39.588+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:39.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:39.588+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:39.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:39.610+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:39.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:39.610+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:39.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:39.610+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:40.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:54:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:54:40.080+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 203 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:54:40.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:40.579+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:40.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:40.579+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:40.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:40.579+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:40.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:40.555+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:40.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:40.555+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:40.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:40.555+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:40.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:40.588+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:40.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:40.588+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:40.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:40.588+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:41.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:41.579+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:41.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:41.579+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:41.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:41.579+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:41.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:41.548+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:41.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:41.548+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:41.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:41.548+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:41.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:41.561+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:41.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:41.561+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:41.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:41.561+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:42.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:42.572+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:42.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:42.572+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:42.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:42.572+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:42.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:42.577+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:42.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:42.577+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:42.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:42.577+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:42.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:42.537+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:42.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:42.537+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:42.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:42.537+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:43.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:43.596+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:43.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:43.596+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:43.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:43.596+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:43.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:43.552+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:43.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:43.552+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:43.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:43.552+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:43.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:43.510+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:43.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:43.510+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:43.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:43.510+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:44.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:44.620+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:44.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:44.620+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:44.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:44.620+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:44.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:44.529+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:44.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:44.529+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:44.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:44.529+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:44.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:44.542+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:44.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:44.542+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:44.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:44.542+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:45.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:54:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:54:45.080+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 234 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:54:45.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:45.580+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:45.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:45.580+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:45.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:45.580+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:45.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:45.556+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:45.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:45.556+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:45.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:45.556+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:45.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:45.591+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:45.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:45.591+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:45.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:45.591+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:46.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:46.584+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:46.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:46.584+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:46.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:46.584+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:46.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:46.543+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:46.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:46.543+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:46.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:46.543+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:46.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:46.593+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:46.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:46.593+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:46.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:46.593+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:47.872 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:47.630+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:47.872 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:47.630+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:47.872 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:47.630+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:47.873 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:47.507+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:47.873 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:47.507+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:47.873 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:47.507+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:47.873 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:47.566+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:47.873 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:47.566+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:47.873 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:47.566+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:48.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:48.538+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:48.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:48.538+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:48.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:48.538+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:48.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:48.621+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:48.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:48.621+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:48.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:48.621+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:48.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:48.564+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:48.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:48.564+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:48.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:48.564+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:49.570 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:49.489+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:49.570 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:49.489+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:49.570 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:49.489+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:49.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:49.588+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:49.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:49.588+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:49.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:49.588+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:49.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:49.572+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:49.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:49.572+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:49.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:49.572+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:50.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:54:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:54:50.081+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 261 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:54:50.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:50.620+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:50.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:50.620+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:50.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:50.620+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:50.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:50.539+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:50.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:50.539+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:50.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:50.539+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:50.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:50.608+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:50.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:50.608+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:50.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:50.608+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:51.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:51.581+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:51.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:51.581+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:51.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:51.581+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:51.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:51.511+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:51.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:51.511+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:51.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:51.511+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:51.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:51.606+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:51.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:51.606+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:51.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:51.606+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:52.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:52.606+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:52.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:52.606+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:52.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:52.606+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:52.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:52.561+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:52.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:52.561+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:52.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:52.561+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:52.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:52.595+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:52.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:52.595+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:52.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:52.595+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:53.955 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:53.586+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:53.955 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:53.586+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:53.955 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:53.586+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:53.955 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:53.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:53.956 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:53.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:53.956 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:53.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:53.956 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:53.602+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:53.956 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:53.602+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:53.956 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:53.602+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:53.956 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:53.602+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14480.14:476 2.d 2:b02ffd62:::200.00000010:head [write 4175421~1452 in=1452b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:54:54.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:54.603+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:54.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:54.603+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:54.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:54.603+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:54.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:54.561+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:54.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:54.561+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:54.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:54.561+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:54.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:54.561+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14480.14:476 2.d 2:b02ffd62:::200.00000010:head [write 4175421~1452 in=1452b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:54:54.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:54.634+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:54.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:54.634+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:54.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:54.634+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:55.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:54:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:54:55.081+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 295 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:54:55.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:55.643+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:55.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:55.643+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:55.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:55.643+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:55.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:55.526+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:55.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:55.526+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:55.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:55.526+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:55.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:55.526+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14480.14:476 2.d 2:b02ffd62:::200.00000010:head [write 4175421~1452 in=1452b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:54:55.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:55.597+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:55.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:55.597+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:55.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:55.597+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:56.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:56.641+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:56.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:56.641+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:56.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:56.641+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:56.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:56.642+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:54:56.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:56.559+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:56.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:56.559+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:56.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:56.559+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:56.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:56.560+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:54:56.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:56.564+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:56.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:56.564+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:56.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:56.564+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:56.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:56.566+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:54:57.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:57.572+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:57.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:57.572+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:57.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:57.572+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:57.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:57.573+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:54:57.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:57.673+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:57.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:57.673+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:57.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:57.673+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:57.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:57.673+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:54:57.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:57.554+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:57.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:57.554+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:57.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:57.554+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:57.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:57.555+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:54:58.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:58.579+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:58.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:58.579+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:58.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:58.579+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:58.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:58.579+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:54:58.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:58.647+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:58.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:58.647+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:58.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:58.647+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:58.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:58.647+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:54:58.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:58.556+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:58.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:58.556+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:58.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:58.556+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:58.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:58.556+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:54:59.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:59.602+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:59.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:59.602+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:59.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:59.602+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:54:59.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:54:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:54:59.603+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:54:59.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:59.651+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:59.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:59.651+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:59.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:59.652+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:54:59.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:54:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:54:59.653+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:54:59.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:59.531+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:59.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:59.531+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:59.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:59.531+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:54:59.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:54:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:54:59.532+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:00.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:55:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:55:00.083+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 323 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:55:00.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:00.548+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:00.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:00.548+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:00.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:00.548+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:00.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:00.548+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:00.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:00.685+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:00.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:00.685+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:00.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:00.685+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:00.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:00.686+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:00.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:00.608+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:00.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:00.608+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:00.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:00.608+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:00.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:00.609+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:01.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:01.619+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:01.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:01.619+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:01.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:01.619+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:01.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:01.619+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:01.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:01.688+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:01.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:01.688+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:01.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:01.688+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:01.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:01.689+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:01.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:01.580+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:01.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:01.580+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:01.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:01.580+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:01.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:01.580+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:02.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:02.734+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:02.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:02.734+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:02.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:02.734+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:02.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:02.735+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:02.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:02.536+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:02.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:02.536+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:02.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:02.536+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:02.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:02.537+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:02.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:02.650+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:02.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:02.650+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:02.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:02.650+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:02.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:02.651+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:03.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:03.519+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:03.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:03.519+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:03.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:03.519+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:03.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:03.520+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:03.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:03.686+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:03.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:03.686+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:03.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:03.686+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:03.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:03.687+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:03.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:03.638+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:03.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:03.638+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:03.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:03.638+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:03.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:03.639+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:04.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:04.517+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:04.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:04.517+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:04.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:04.517+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:04.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:04.518+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:04.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:04.640+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:04.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:04.640+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:04.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:04.640+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:04.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:04.640+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:04.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:04.636+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:04.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:04.636+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:04.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:04.636+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:04.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:04.636+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:05.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:55:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:55:05.083+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 353 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:55:05.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:05.662+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:05.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:05.662+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:05.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:05.662+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:05.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:05.663+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:05.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:05.541+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:05.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:05.541+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:05.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:05.541+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:05.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:05.542+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:05.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:05.652+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:05.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:05.652+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:05.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:05.652+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:05.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:05.653+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:06.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:06.692+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:06.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:06.692+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:06.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:06.692+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:06.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:06.693+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:06.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:06.637+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:06.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:06.637+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:06.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:06.637+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:06.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:06.639+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:06.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:06.512+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:06.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:06.512+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:06.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:06.512+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:06.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:06.513+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:07.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:07.692+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:07.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:07.692+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:07.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:07.692+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:07.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:07.692+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:07.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:07.615+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:07.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:07.615+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:07.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:07.615+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:07.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:07.615+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:07.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:07.549+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:07.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:07.549+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:07.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:07.549+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:07.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:07.550+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:08.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:08.662+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:08.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:08.662+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:08.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:08.662+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:08.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:08.663+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:08.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:08.712+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:08.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:08.712+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:08.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:08.712+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:08.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:08.713+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:08.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:08.524+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:08.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:08.524+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:08.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:08.524+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:08.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:08.525+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:09.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:09.759+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:09.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:09.759+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:09.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:09.759+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:09.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:09.760+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:09.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:09.669+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:09.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:09.669+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:09.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:09.669+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:09.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:09.671+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:09.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:09.538+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:09.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:09.538+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:09.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:09.538+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:09.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:09.539+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:10.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:55:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:55:10.084+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 384 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:55:10.788 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:10.674+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:10.788 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:10.674+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:10.788 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:10.674+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:10.788 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:10.675+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:10.788 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:10.514+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:10.789 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:10.514+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:10.789 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:10.514+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:10.789 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:10.515+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:11.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:10.791+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:11.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:10.791+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:11.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:10.791+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:11.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:10.791+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:11.831 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:11.544+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:11.831 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:11.544+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:11.831 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:11.544+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:11.831 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:11.545+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:11.832 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:11.636+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:11.832 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:11.636+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:11.832 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:11.636+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:11.832 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:11.637+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:12.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:11.834+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:12.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:11.834+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:12.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:11.834+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:12.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:11.835+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:12.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:12.597+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:12.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:12.597+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:12.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:12.597+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:12.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:12.598+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:12.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:12.568+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:12.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:12.568+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:12.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:12.568+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:12.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:12.569+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:13.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:12.837+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:13.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:12.837+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:13.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:12.837+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:13.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:12.838+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:13.864 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:13.594+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:13.865 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:13.594+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:13.865 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:13.594+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:13.865 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:13.595+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:13.865 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:13.606+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:13.865 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:13.606+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:13.865 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:13.606+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:13.865 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:13.607+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:14.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:13.867+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:14.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:13.867+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:14.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:13.867+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:14.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:13.868+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:14.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:14.597+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:14.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:14.597+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:14.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:14.597+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:14.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:14.597+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:14.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:14.830+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:14.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:14.830+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:14.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:14.830+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:14.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:14.830+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:14.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:14.614+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:14.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:14.614+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:14.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:14.614+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:14.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:14.615+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:15.459 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:55:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:55:15.084+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 416 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:55:15.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:15.594+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:15.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:15.594+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:15.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:15.594+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:15.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:15.595+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:15.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:15.570+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:15.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:15.570+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:15.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:15.570+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:15.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:15.571+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:15.804+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:15.804+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:15.804+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:15.805+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:16.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:16.606+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:16.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:16.606+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:16.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:16.606+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:16.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:16.607+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:16.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:16.791+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:16.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:16.791+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:16.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:16.791+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:16.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:16.791+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:16.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:16.638+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:16.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:16.638+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:16.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:16.638+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:16.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:16.639+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:17.873 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:17.816+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:17.873 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:17.816+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:17.873 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:17.816+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:17.873 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:17.817+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:17.873 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:17.567+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:17.874 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:17.567+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:17.874 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:17.567+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:17.874 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:17.568+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:17.874 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:17.612+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:17.874 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:17.612+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:17.874 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:17.612+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:17.874 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:17.613+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:18.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:18.645+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:18.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:18.645+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:18.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:18.645+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:18.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:18.646+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:18.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:18.778+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:18.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:18.778+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:18.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:18.778+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:18.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:18.779+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:18.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:18.560+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:18.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:18.560+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:18.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:18.560+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:18.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:18.561+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:19.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:19.546+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:19.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:19.546+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:19.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:19.546+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:19.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:19.547+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:19.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:19.733+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:19.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:19.733+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:19.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:19.733+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:19.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:19.733+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:19.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:19.684+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:19.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:19.684+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:19.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:19.684+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:19.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:19.685+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:20.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:55:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:55:20.085+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 445 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:55:20.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:20.771+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:20.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:20.771+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:20.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:20.771+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:20.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:20.772+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:20.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:20.535+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:20.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:20.535+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:20.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:20.535+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:20.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:20.536+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:20.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:20.706+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:20.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:20.706+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:20.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:20.706+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:20.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:20.707+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:21.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:21.528+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:21.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:21.528+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:21.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:21.528+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:21.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:21.529+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:21.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:21.769+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:21.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:21.769+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:21.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:21.769+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:21.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:21.770+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:21.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:21.682+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:21.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:21.682+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:21.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:21.682+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:21.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:21.683+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:22.811 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:22.535+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:22.811 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:22.535+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:22.811 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:22.535+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:22.811 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:22.536+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:22.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:22.665+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:22.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:22.665+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:22.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:22.665+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:22.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:22.666+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:23.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:22.814+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:23.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:22.814+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:23.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:22.814+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:23.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:22.815+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:23.832 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:23.559+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:23.832 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:23.559+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:23.832 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:23.559+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:23.832 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:23.561+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:23.833 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:23.652+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:23.833 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:23.652+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:23.833 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:23.652+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:23.833 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:23.653+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:24.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:23.835+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:24.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:23.835+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:24.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:23.835+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:24.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:23.836+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:24.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:24.665+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:24.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:24.665+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:24.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:24.665+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:24.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:24.665+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:24.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:24.582+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:24.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:24.582+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:24.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:24.582+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:24.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:24.583+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:24.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:24.816+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:24.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:24.816+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:24.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:24.816+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:24.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:24.817+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:25.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:55:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:55:25.085+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 480 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:55:25.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:25.641+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:25.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:25.641+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:25.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:25.641+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:25.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:25.642+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:25.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:25.769+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:25.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:25.769+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:25.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:25.769+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:25.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:25.770+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:25.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:25.548+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:25.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:25.548+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:25.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:25.548+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:25.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:25.550+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:26.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:26.589+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:26.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:26.589+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:26.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:26.589+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:26.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:26.590+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:26.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:26.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:26.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:26.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:26.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:26.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:26.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:26.627+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:26.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:26.772+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:26.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:26.772+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:26.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:26.772+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:26.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:26.773+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:27.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:27.590+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:27.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:27.590+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:27.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:27.590+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:27.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:27.591+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:27.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:27.781+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:27.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:27.781+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:27.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:27.781+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:27.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:27.782+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:27.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:27.647+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:27.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:27.647+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:27.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:27.647+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:27.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:27.648+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:28.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:28.749+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:28.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:28.750+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:28.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:28.750+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:28.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:28.750+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:28.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:28.601+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:28.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:28.601+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:28.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:28.601+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:28.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:28.602+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:28.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:28.612+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:28.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:28.612+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:28.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:28.612+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:28.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:28.613+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:29.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:29.701+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:29.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:29.701+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:29.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:29.701+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:29.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:29.702+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:29.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:29.568+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:29.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:29.568+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:29.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:29.568+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:29.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:29.569+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:29.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:29.645+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:29.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:29.645+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:29.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:29.645+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:29.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:29.646+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:30.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:55:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:55:30.086+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 512 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:55:30.748 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:30.595+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:30.748 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:30.595+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:30.749 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:30.595+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:30.749 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:30.596+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:30.749 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:30.607+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:30.749 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:30.607+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:30.749 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:30.607+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:30.749 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:30.608+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:31.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:30.751+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:31.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:30.751+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:31.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:30.751+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:31.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:30.753+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:31.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:31.747+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:31.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:31.747+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:31.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:31.747+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:31.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:31.748+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:31.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:31.618+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:31.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:31.618+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:31.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:31.618+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:31.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:31.619+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:31.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:31.632+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:31.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:31.632+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:31.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:31.632+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:31.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:31.633+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:32.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:32.722+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:32.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:32.722+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:32.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:32.722+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:32.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:32.723+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:32.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:32.619+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:32.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:32.619+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:32.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:32.619+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:32.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:32.620+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:32.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:32.579+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:32.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:32.579+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:32.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:32.579+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:32.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:32.580+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:33.891 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:33.681+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:33.891 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:33.681+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:33.891 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:33.681+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:33.891 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:33.681+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:33.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:33.536+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:33.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:33.536+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:33.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:33.536+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:33.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:33.537+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:33.892 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:33.609+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:33.892 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:33.609+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:33.892 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:33.609+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:33.893 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:33.609+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:34.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:34.629+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:34.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:34.629+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:34.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:34.629+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:34.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:34.630+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:34.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:34.656+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:34.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:34.656+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:34.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:34.656+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:34.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:34.657+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:34.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:34.551+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:34.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:34.551+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:34.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:34.551+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:34.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:34.551+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:35.470 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:55:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:55:35.087+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 541 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:55:35.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:35.641+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:35.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:35.641+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:35.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:35.641+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:35.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:35.642+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:35.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:35.599+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:35.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:35.599+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:35.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:35.599+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:35.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:35.600+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:35.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:35.554+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:35.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:35.554+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:35.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:35.554+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:35.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:35.555+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:36.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:36.663+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:36.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:36.663+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:36.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:36.663+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:36.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:36.664+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:36.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:36.575+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:36.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:36.575+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:36.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:36.575+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:36.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:36.576+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:36.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:36.575+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:36.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:36.575+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:36.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:36.575+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:36.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:36.576+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:37.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:37.622+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:37.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:37.622+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:37.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:37.622+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:37.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:37.623+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:37.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:37.556+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:37.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:37.556+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:37.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:37.556+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:37.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:37.556+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:37.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:37.553+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:37.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:37.553+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:37.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:37.553+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:37.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:37.554+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:38.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:38.601+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:38.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:38.601+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:38.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:38.601+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:38.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:38.602+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:38.585+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:38.585+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:38.585+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:38.586+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:38.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:38.590+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:38.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:38.590+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:38.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:38.591+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:38.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:38.591+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:39.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:39.591+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:39.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:39.591+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:39.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:39.591+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:39.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:39.592+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:39.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:39.615+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:39.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:39.615+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:39.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:39.615+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:39.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:39.615+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:39.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:39.578+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:39.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:39.578+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:39.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:39.578+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:39.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:39.578+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:40.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:55:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:55:40.087+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 570 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:55:40.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:40.590+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:40.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:40.590+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:40.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:40.590+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:40.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:40.591+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:40.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:40.605+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:40.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:40.605+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:40.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:40.605+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:40.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:40.606+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:40.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:40.598+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:40.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:40.598+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:40.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:40.599+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:40.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:40.600+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:41.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:41.573+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:41.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:41.573+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:41.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:41.573+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:41.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:41.574+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:41.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:41.554+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:41.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:41.554+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:41.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:41.554+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:41.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:41.555+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:41.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:41.590+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:41.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:41.590+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:41.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:41.591+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:41.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:41.592+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:42.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:42.621+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:42.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:42.621+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:42.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:42.621+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:42.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:42.622+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:42.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:42.568+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:42.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:42.568+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:42.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:42.568+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:42.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:42.568+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:42.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:42.514+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:42.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:42.514+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:42.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:42.514+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:42.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:42.515+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:43.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:43.623+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:43.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:43.623+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:43.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:43.623+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:43.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:43.624+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:43.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:43.570+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:43.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:43.570+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:43.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:43.570+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:43.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:43.570+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:43.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:43.486+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:43.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:43.486+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:43.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:43.486+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:43.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:43.486+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:44.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:44.617+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:44.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:44.617+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:44.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:44.617+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:44.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:44.618+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:44.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:44.503+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:44.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:44.503+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:44.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:44.503+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:44.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:44.504+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:44.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:44.596+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:44.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:44.596+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:44.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:44.596+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:44.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:44.596+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:45.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:55:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:55:45.087+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 600 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:55:45.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:45.522+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:45.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:45.522+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:45.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:45.522+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:45.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:45.522+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:45.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:45.558+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:45.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:45.558+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:45.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:45.558+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:45.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:45.559+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:45.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:45.611+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:45.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:45.611+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:45.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:45.611+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:45.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:45.611+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:46.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:46.561+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:46.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:46.561+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:46.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:46.561+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:46.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:46.562+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:46.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:46.562+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:46.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:46.562+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:46.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:46.562+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:46.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:46.562+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:46.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:46.550+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:46.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:46.550+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:46.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:46.550+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:46.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:46.550+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:47.872 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:47.523+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:47.872 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:47.523+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:47.873 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:47.523+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:47.873 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:47.523+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:47.873 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:47.604+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:47.873 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:47.604+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:47.873 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:47.604+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:47.873 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:47.605+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:47.873 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:47.564+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:47.873 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:47.564+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:47.873 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:47.564+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:47.874 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:47.565+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:48.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:48.550+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:48.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:48.551+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:48.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:48.551+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:48.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:48.551+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:48.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:48.615+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:48.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:48.615+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:48.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:48.615+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:48.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:48.616+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:48.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:48.545+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:48.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:48.545+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:48.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:48.545+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:48.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:48.546+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:49.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:49.648+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:49.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:49.648+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:49.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:49.648+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:49.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:49.648+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:49.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:49.536+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:49.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:49.536+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:49.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:49.536+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:49.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:49.537+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:49.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:49.575+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:49.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:49.575+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:49.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:49.575+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:49.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:49.575+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:50.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:55:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:55:50.088+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 628 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:55:50.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:50.517+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:50.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:50.517+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:50.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:50.517+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:50.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:50.517+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:50.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:50.613+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:50.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:50.613+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:50.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:50.613+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:50.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:50.614+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:50.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:50.561+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:50.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:50.561+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:50.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:50.561+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:50.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:50.562+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:51.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:51.587+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:51.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:51.587+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:51.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:51.587+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:51.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:51.588+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:51.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:51.610+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:51.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:51.610+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:51.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:51.610+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:51.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:51.610+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:51.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:51.534+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:51.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:51.534+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:51.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:51.534+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:51.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:51.535+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:52.628 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:52.534+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:52.628 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:52.534+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:52.628 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:52.534+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:52.628 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:52.535+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:52.629 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:52.545+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:52.629 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:52.545+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:52.629 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:52.545+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:52.629 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:52.545+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:52.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:52.631+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:52.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:52.631+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:52.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:52.631+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:52.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:52.632+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:53.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:53.537+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:53.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:53.537+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:53.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:53.538+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:53.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:53.538+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:53.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:53.500+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:53.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:53.500+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:53.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:53.500+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:53.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:53.501+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:53.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:53.636+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:53.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:53.636+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:53.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:53.636+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:53.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:53.636+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:54.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:54.622+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:54.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:54.622+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:54.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:54.622+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:54.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:54.623+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:54.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:54.510+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:54.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:54.510+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:54.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:54.510+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:54.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:54.511+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:54.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:54.559+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:54.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:54.559+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:54.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:54.559+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:54.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:54.559+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:55.395 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:55:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:55:55.088+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 657 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:55:55.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:55.552+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:55.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:55.552+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:55.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:55.552+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:55.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:55.553+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:55.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:55.547+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:55.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:55.547+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:55.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:55.547+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:55.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:55.547+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:55.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:55.592+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:55.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:55.592+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:55.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:55.592+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:55.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:55.593+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:56.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:56.591+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:56.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:56.591+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:56.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:56.591+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:56.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:56.592+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:56.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:56.624+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:56.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:56.624+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:56.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:56.624+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:56.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:56.625+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:56.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:56.508+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:56.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:56.508+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:56.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:56.508+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:56.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:56.509+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:57.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:57.479+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:57.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:57.479+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:57.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:57.479+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:57.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:57.479+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:57.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:57.627+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:57.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:57.627+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:57.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:57.627+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:57.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:57.628+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:57.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:57.600+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:57.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:57.600+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:57.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:57.600+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:57.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:57.601+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:58.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:58.651+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:58.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:58.651+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:58.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:58.651+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:58.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:58.652+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:58.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:58.621+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:58.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:58.621+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:58.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:58.621+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:58.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:58.622+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:58.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:58.451+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:58.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:58.451+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:58.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:58.451+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:58.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:58.452+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:59.637 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:59.493+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:59.638 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:59.493+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:59.638 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:59.493+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:55:59.638 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:55:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:55:59.494+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:55:59.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:59.655+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:59.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:59.655+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:59.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:59.655+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:55:59.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:55:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:55:59.656+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:55:59.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:59.639+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:59.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:59.639+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:59.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:59.639+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:55:59.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:55:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:55:59.640+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:00.466 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:56:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:56:00.088+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 686 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:56:00.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:00.644+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:00.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:00.644+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:00.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:00.644+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:00.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:00.645+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:00.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:00.468+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:00.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:00.468+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:00.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:00.468+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:00.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:00.468+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:00.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:00.665+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:00.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:00.665+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:00.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:00.665+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:00.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:00.665+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:01.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:01.657+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:01.682 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:01.657+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:01.682 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:01.657+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:01.682 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:01.657+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:01.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:01.421+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:01.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:01.421+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:01.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:01.421+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:01.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:01.422+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:01.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:01.683+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:01.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:01.683+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:01.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:01.683+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:01.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:01.684+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:02.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:02.449+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:02.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:02.449+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:02.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:02.449+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:02.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:02.450+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:02.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:02.667+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:02.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:02.667+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:02.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:02.667+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:02.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:02.667+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:03.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:02.730+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:03.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:02.730+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:03.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:02.730+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:03.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:02.731+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:03.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:03.642+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:03.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:03.642+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:03.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:03.642+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:03.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:03.643+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:03.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:03.428+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:03.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:03.428+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:03.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:03.428+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:03.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:03.429+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:04.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:03.771+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:04.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:03.771+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:04.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:03.771+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:04.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:03.772+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:04.648 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:04.399+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:04.648 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:04.399+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:04.649 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:04.399+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:04.649 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:04.400+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:04.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:04.650+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:04.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:04.650+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:04.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:04.650+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:04.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:04.651+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:04.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:04.727+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:04.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:04.727+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:04.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:04.727+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:04.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:04.727+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:05.366 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:56:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:56:05.088+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 718 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:56:05.630 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:05.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:05.630 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:05.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:05.630 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:05.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:05.630 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:05.369+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:05.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:05.632+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:05.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:05.632+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:05.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:05.632+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:05.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:05.632+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:05.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:05.723+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:05.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:05.723+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:05.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:05.723+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:05.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:05.724+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:06.372+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:06.372+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:06.372+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:06.373+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:06.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:06.598+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:06.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:06.598+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:06.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:06.598+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:06.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:06.599+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:07.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:06.773+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:07.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:06.773+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:07.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:06.773+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:07.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:06.774+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:07.685 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:07.373+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:07.685 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:07.373+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:07.685 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:07.373+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:07.685 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:07.374+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:07.685 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:07.575+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:07.685 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:07.575+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:07.685 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:07.575+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:07.685 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:07.576+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:08.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:07.754+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:08.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:07.754+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:08.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:07.754+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:08.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:07.755+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:08.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:08.378+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:08.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:08.378+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:08.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:08.378+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:08.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:08.378+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:08.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:08.590+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:08.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:08.590+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:08.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:08.590+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:08.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:08.591+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:09.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:08.741+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:09.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:08.741+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:09.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:08.741+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:09.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:08.742+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:09.577 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:09.335+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:09.578 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:09.335+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:09.578 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:09.335+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:09.578 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:09.336+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:09.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:09.747+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:09.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:09.747+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:09.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:09.747+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:09.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:09.748+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:09.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:09.579+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:09.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:09.579+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:09.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:09.579+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:09.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:09.581+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:10.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:56:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:56:10.089+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 746 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:56:10.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:10.294+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:10.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:10.294+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:10.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:10.294+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:10.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:10.295+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:10.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:10.765+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:10.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:10.765+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:10.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:10.765+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:10.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:10.766+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:10.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:10.582+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:10.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:10.582+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:10.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:10.582+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:10.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:10.583+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:11.632 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:11.322+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:11.632 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:11.322+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:11.632 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:11.322+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:11.632 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:11.323+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:11.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:11.557+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:11.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:11.557+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:11.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:11.557+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:11.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:11.558+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:11.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:11.737+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:11.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:11.737+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:11.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:11.737+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:11.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:11.738+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:12.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:12.370+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:12.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:12.370+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:12.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:12.370+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:12.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:12.371+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:12.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:12.605+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:12.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:12.605+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:12.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:12.605+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:12.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:12.605+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:13.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:12.766+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:13.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:12.766+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:13.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:12.766+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:13.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:12.767+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:13.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:13.407+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:13.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:13.407+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:13.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:13.407+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:13.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:13.408+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:13.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:13.625+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:13.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:13.625+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:13.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:13.625+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:13.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:13.626+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:14.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:13.783+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:14.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:13.783+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:14.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:13.783+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:14.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:13.784+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:14.670 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:14.395+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:14.670 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:14.395+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:14.670 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:14.395+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:14.670 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:14.395+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:14.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:14.735+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:14.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:14.735+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:14.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:14.735+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:14.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:14.736+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:14.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:14.672+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:14.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:14.672+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:14.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:14.672+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:14.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:14.673+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:15.385 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:56:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:56:15.089+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 790 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:56:15.687 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:15.387+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:15.687 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:15.387+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:15.687 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:15.387+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:15.687 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:15.388+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:15.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:15.690+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:15.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:15.690+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:15.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:15.690+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:15.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:15.690+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:15.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:15.750+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:15.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:15.750+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:15.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:15.750+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:15.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:15.750+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:16.710 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:16.429+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:16.710 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:16.429+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:16.711 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:16.429+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:16.711 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:16.430+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:16.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:16.712+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:16.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:16.712+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:16.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:16.712+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:16.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:16.713+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:16.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:16.789+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:16.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:16.789+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:16.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:16.789+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:16.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:16.789+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:17.710 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:17.385+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:17.710 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:17.385+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:17.710 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:17.385+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:17.710 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:17.385+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:17.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:17.778+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:17.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:17.778+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:17.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:17.778+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:17.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:17.779+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:17.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:17.713+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:17.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:17.713+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:17.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:17.713+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:17.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:17.713+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:18.675 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:18.355+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:18.675 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:18.355+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:18.675 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:18.355+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:18.675 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:18.356+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:18.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:18.807+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:18.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:18.807+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:18.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:18.807+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:18.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:18.808+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:18.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:18.678+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:18.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:18.678+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:18.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:18.678+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:18.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:18.679+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:19.637 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:19.328+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:19.637 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:19.328+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:19.637 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:19.328+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:19.637 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:19.329+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:19.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:19.792+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:19.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:19.792+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:19.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:19.792+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:19.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:19.793+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:19.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:19.640+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:19.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:19.640+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:19.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:19.640+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:19.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:19.641+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:20.327 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:56:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:56:20.091+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 819 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:56:20.613 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:20.331+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:20.613 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:20.331+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:20.613 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:20.331+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:20.613 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:20.331+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:20.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:20.833+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:20.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:20.833+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:20.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:20.833+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:20.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:20.834+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:20.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:20.616+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:20.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:20.616+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:20.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:20.616+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:20.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:20.617+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:21.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:21.571+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:21.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:21.571+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:21.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:21.571+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:21.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:21.572+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:21.633 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:21.369+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:21.633 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:21.369+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:21.633 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:21.369+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:21.634 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:21.369+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:21.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:21.861+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:21.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:21.861+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:21.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:21.861+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:21.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:21.862+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:22.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:22.607+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:22.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:22.607+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:22.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:22.607+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:22.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:22.608+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:22.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:22.359+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:22.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:22.359+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:22.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:22.359+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:22.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:22.359+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:23.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:22.830+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:23.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:22.830+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:23.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:22.830+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:23.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:22.831+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:23.570 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:23.311+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:23.570 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:23.311+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:23.570 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:23.311+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:23.570 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:23.311+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:23.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:23.792+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:23.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:23.792+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:23.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:23.792+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:23.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:23.792+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:23.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:23.573+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:23.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:23.573+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:23.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:23.573+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:23.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:23.573+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:24.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:24.324+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:24.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:24.324+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:24.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:24.324+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:24.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:24.325+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:24.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:24.594+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:24.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:24.594+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:24.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:24.594+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:24.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:24.595+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:24.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:24.816+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:24.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:24.816+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:24.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:24.816+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:24.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:24.817+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:25.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:56:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:56:25.091+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 849 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:56:25.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:25.284+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:25.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:25.284+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:25.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:25.284+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:25.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:25.285+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:25.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:25.780+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:25.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:25.780+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:25.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:25.780+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:25.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:25.781+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:25.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:25.579+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:25.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:25.579+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:25.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:25.579+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:25.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:25.580+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:26.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:26.296+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:26.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:26.296+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:26.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:26.296+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:26.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:26.297+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:26.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:26.797+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:26.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:26.797+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:26.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:26.797+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:26.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:26.797+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:26.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:26.604+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:26.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:26.604+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:26.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:26.604+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:26.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:26.604+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:27.560 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:27.254+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:27.560 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:27.254+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:27.560 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:27.254+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:27.561 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:27.255+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:27.815 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:27.563+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:27.815 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:27.563+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:27.815 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:27.563+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:27.815 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:27.564+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:28.215 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:27.818+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:28.216 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:27.818+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:28.216 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:27.818+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:28.216 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:27.819+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:28.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:28.218+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:28.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:28.218+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:28.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:28.218+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:28.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:28.219+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:28.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:28.540+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:28.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:28.540+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:28.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:28.540+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:28.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:28.541+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:28.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:28.772+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:28.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:28.772+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:28.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:28.772+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:28.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:28.773+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:29.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:29.219+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:29.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:29.219+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:29.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:29.219+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:29.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:29.220+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:29.771 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:29.496+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:29.772 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:29.496+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:29.772 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:29.496+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:29.772 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:29.497+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:30.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:29.774+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:30.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:29.774+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:30.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:29.774+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:30.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:29.775+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:30.470 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:56:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:56:30.092+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 877 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:56:30.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:30.217+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:30.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:30.217+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:30.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:30.217+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:30.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:30.218+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:30.774 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:30.513+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:30.774 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:30.513+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:30.774 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:30.513+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:30.774 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:30.514+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:31.212 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:30.777+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:31.212 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:30.777+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:31.212 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:30.777+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:31.212 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:30.778+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:31.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:31.215+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:31.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:31.215+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:31.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:31.215+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:31.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:31.215+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:31.765 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:31.495+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:31.765 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:31.495+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:31.765 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:31.495+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:31.765 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:31.496+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:32.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:31.768+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:32.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:31.768+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:32.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:31.768+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:32.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:31.769+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:32.534 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:32.247+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:32.534 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:32.247+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:32.534 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:32.247+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:32.534 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:32.247+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:32.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:32.537+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:32.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:32.537+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:32.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:32.537+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:32.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:32.537+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:32.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:32.766+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:32.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:32.766+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:32.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:32.766+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:32.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:32.767+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:33.578 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:33.291+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:33.578 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:33.291+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:33.578 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:33.291+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:33.579 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:33.292+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:33.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:33.721+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:33.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:33.721+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:33.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:33.721+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:33.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:33.722+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:33.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:33.581+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:33.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:33.581+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:33.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:33.581+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:33.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:33.582+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:34.620 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:34.255+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:34.620 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:34.255+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:34.620 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:34.255+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:34.620 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:34.256+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:34.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:34.769+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:34.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:34.769+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:34.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:34.769+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:34.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:34.769+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:34.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:34.623+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:34.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:34.623+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:34.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:34.623+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:34.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:34.624+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:35.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:56:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:56:35.093+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 907 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:56:35.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:35.252+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:35.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:35.252+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:35.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:35.252+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:35.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:35.253+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:35.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:35.749+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:35.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:35.749+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:35.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:35.749+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:35.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:35.750+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:35.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:35.612+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:35.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:35.612+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:35.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:35.612+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:35.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:35.613+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:36.604 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:36.277+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:36.604 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:36.277+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:36.604 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:36.277+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:36.604 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:36.277+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:36.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:36.607+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:36.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:36.607+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:36.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:36.607+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:36.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:36.608+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:36.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:36.757+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:36.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:36.757+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:36.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:36.757+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:36.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:36.758+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:37.577 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:37.249+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:37.577 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:37.249+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:37.577 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:37.249+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:37.577 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:37.250+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:37.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:37.738+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:37.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:37.738+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:37.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:37.738+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:37.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:37.739+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:37.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:37.580+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:37.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:37.580+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:37.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:37.580+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:37.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:37.581+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:38.594 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:38.275+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:38.594 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:38.275+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:38.594 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:38.275+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:38.594 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:38.276+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:38.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:38.701+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:38.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:38.701+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:38.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:38.701+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:38.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:38.702+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:38.597+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:38.597+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:38.597+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:38.598+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:39.636 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:39.279+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:39.637 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:39.279+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:39.637 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:39.279+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:39.637 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:39.280+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:39.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:39.675+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:39.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:39.675+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:39.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:39.675+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:39.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:39.676+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:39.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:39.639+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:39.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:39.639+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:39.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:39.639+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:39.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:39.640+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:40.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:56:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:56:40.093+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 935 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:56:40.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:40.274+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:40.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:40.274+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:40.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:40.274+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:40.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:40.274+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:40.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:40.663+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:40.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:40.663+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:40.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:40.663+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:40.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:40.663+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:40.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:40.718+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:40.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:40.718+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:40.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:40.718+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:40.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:40.719+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:41.635 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:41.305+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:41.635 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:41.305+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:41.635 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:41.305+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:41.635 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:41.306+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:41.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:41.681+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:41.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:41.681+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:41.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:41.681+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:41.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:41.682+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:41.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:41.721+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:41.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:41.721+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:41.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:41.721+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:41.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:41.722+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:42.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:42.348+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:42.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:42.348+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:42.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:42.348+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:42.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:42.349+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:42.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:42.758+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:42.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:42.758+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:42.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:42.758+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:42.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:42.759+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:42.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:42.684+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:42.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:42.684+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:42.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:42.685+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:42.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:42.686+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:43.651 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:43.323+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:43.651 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:43.323+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:43.651 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:43.323+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:43.651 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:43.324+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:43.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:43.731+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:43.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:43.731+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:43.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:43.731+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:43.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:43.732+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:43.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:43.654+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:43.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:43.654+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:43.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:43.654+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:43.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:43.655+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:44.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:44.279+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:44.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:44.279+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:44.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:44.279+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:44.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:44.280+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:44.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:44.765+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:44.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:44.765+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:44.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:44.765+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:44.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:44.765+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:44.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:44.627+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:44.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:44.627+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:44.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:44.627+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:44.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:44.628+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:45.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:56:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:56:45.094+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 965 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:56:45.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:45.246+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:45.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:45.246+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:45.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:45.246+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:45.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:45.246+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:45.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:45.800+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:45.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:45.800+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:45.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:45.800+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:45.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:45.801+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:45.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:45.635+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:45.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:45.635+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:45.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:45.635+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:45.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:45.636+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:46.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:46.211+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:46.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:46.211+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:46.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:46.211+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:46.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:46.212+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:46.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:46.770+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:46.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:46.770+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:46.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:46.770+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:46.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:46.771+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:46.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:46.684+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:46.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:46.684+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:46.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:46.684+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:46.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:46.684+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:47.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:47.180+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:47.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:47.180+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:47.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:47.180+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:47.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:47.180+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:47.789 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:47.705+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:47.789 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:47.705+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:47.789 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:47.705+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:47.789 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:47.706+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:48.145 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:47.792+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:48.145 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:47.792+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:48.145 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:47.792+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:48.145 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:47.793+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:48.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:48.148+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:48.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:48.148+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:48.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:48.148+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:48.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:48.148+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:49.173 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:48.753+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:49.173 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:48.753+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:49.173 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:48.753+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:49.173 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:48.753+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:49.173 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:48.751+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:49.173 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:48.751+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:49.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:48.751+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:49.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:48.752+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:49.174 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:49.176+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:49.174 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:49.176+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:49.174 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:49.176+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:49.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:49.176+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:50.091 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:49.783+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:50.091 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:49.783+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:50.091 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:49.783+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:50.092 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:49.783+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:50.092 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:49.739+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:50.092 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:49.739+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:50.092 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:49.739+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:50.092 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:49.739+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:50.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:56:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:56:50.094+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 993 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:56:50.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:50.177+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:50.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:50.178+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:50.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:50.178+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:50.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:50.178+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:50.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:50.713+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:50.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:50.713+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:50.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:50.713+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:50.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:50.714+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:50.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:50.775+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:50.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:50.775+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:50.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:50.775+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:50.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:50.775+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:51.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:51.185+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:51.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:51.185+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:51.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:51.185+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:51.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:51.185+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:51.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:51.810+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:51.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:51.810+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:51.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:51.810+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:51.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:51.811+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:51.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:51.704+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:51.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:51.704+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:51.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:51.704+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:51.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:51.705+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:52.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:52.178+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:52.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:52.178+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:52.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:52.178+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:52.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:52.179+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:52.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:52.836+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:52.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:52.836+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:52.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:52.836+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:52.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:52.837+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:52.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:52.694+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:52.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:52.694+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:52.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:52.694+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:52.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:52.695+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:53.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:53.200+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:53.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:53.200+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:53.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:53.200+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:53.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:53.200+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:53.882 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:53.654+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:53.882 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:53.654+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:53.882 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:53.654+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:53.882 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:53.655+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:54.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:53.885+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:54.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:53.885+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:54.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:53.885+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:54.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:53.886+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:54.616 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:54.234+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:54.616 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:54.234+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:54.616 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:54.234+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:54.616 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:54.235+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:54.876 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:54.619+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:54.876 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:54.619+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:54.876 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:54.619+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:54.876 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:54.620+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:55.189 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:56:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:56:55.095+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1026 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:56:55.190 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:54.879+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:55.190 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:54.879+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:55.190 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:54.879+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:55.190 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:54.880+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:55.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:55.193+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:55.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:55.193+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:55.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:55.193+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:55.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:55.193+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:55.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:55.833+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:55.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:55.833+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:55.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:55.833+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:55.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:55.834+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:55.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:55.637+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:55.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:55.637+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:55.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:55.637+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:55.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:55.638+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:56.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:56.210+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:56.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:56.210+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:56.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:56.210+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:56.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:56.211+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:56.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:56.646+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:56.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:56.646+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:56.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:56.646+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:56.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:56.647+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:56.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:56.807+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:56.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:56.807+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:56.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:56.807+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:56.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:56.807+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:57.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:57.178+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:57.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:57.178+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:57.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:57.178+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:57.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:57.179+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:57.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:57.819+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:57.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:57.819+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:57.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:57.819+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:57.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:57.820+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:57.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:57.658+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:57.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:57.658+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:57.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:57.658+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:57.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:57.658+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:58.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:58.189+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:58.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:58.189+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:58.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:58.189+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:58.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:58.190+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:58.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:58.642+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:58.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:58.642+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:58.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:58.642+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:58.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:58.643+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:58.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:58.848+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:58.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:58.848+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:58.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:58.848+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:58.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:58.849+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:56:59.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:59.171+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:59.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:59.171+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:59.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:59.171+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:56:59.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:56:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:56:59.171+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:59.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:59.685+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:59.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:59.686+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:59.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:59.686+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:56:59.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:56:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:56:59.686+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:56:59.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:59.840+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:59.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:59.840+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:59.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:59.840+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:56:59.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:56:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:56:59.840+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:00.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:57:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:57:00.095+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1069 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:57:00.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:00.208+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:00.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:00.208+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:00.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:00.208+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:00.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:00.208+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:00.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:00.682+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:00.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:00.682+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:00.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:00.682+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:00.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:00.683+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:00.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:00.860+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:00.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:00.860+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:00.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:00.860+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:00.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:00.860+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:01.635 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:01.231+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:01.636 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:01.231+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:01.636 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:01.231+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:01.636 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:01.231+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:01.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:01.657+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:01.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:01.657+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:01.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:01.657+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:01.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:01.658+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:01.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:01.815+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:01.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:01.815+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:01.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:01.815+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:01.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:01.815+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:02.693 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:02.257+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:02.693 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:02.257+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:02.693 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:02.257+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:02.693 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:02.257+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:02.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:02.696+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:02.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:02.696+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:02.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:02.696+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:02.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:02.697+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:02.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:02.865+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:02.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:02.865+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:02.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:02.865+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:02.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:02.865+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:03.523 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:03.234+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:03.523 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:03.234+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:03.523 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:03.234+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:03.523 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:03.235+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:03.872 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:03.718+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:03.872 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:03.718+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:03.872 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:03.718+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:03.873 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:03.719+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:04.215 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:03.875+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:04.215 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:03.875+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:04.215 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:03.875+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:04.215 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:03.876+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:04.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:04.218+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:04.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:04.218+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:04.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:04.218+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:04.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:04.219+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:05.093 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:04.902+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:05.094 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:04.902+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:05.094 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:04.902+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:05.094 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:04.903+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:05.094 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:04.726+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:05.094 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:04.726+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:05.094 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:04.726+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:05.094 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:04.727+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:05.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:57:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:57:05.096+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1116 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:57:05.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:05.258+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:05.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:05.258+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:05.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:05.258+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:05.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:05.258+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:05.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:05.702+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:05.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:05.702+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:05.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:05.702+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:05.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:05.703+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:05.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:05.882+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:05.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:05.882+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:05.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:05.882+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:05.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:05.883+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:06.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:06.227+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:06.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:06.227+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:06.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:06.227+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:06.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:06.227+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:07.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:06.732+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:07.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:06.732+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:07.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:06.732+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:07.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:06.733+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:07.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:06.882+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:07.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:06.882+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:07.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:06.882+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:07.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:06.883+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:07.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:07.263+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:07.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:07.263+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:07.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:07.263+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:07.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:07.264+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:08.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:07.768+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:08.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:07.768+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:08.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:07.768+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:08.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:07.769+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:08.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:07.853+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:08.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:07.853+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:08.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:07.853+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:08.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:07.854+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:08.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:08.313+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:08.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:08.313+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:08.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:08.313+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:08.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:08.313+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:09.123 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:08.800+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:09.124 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:08.800+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:09.124 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:08.800+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:09.124 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:08.801+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:09.124 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:08.858+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:09.124 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:08.858+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:09.124 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:08.858+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:09.124 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:08.858+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:09.509 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:09.266+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:09.509 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:09.266+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:09.509 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:09.266+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:09.509 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:09.266+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:10.094 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:09.901+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:10.094 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:09.901+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:10.094 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:09.901+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:10.094 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:09.902+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:10.094 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:09.818+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:10.094 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:09.818+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:10.094 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:09.818+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:10.095 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:09.819+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:10.239 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:57:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:57:10.096+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1161 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:57:10.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:10.241+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:10.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:10.241+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:10.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:10.241+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:10.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:10.242+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:11.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:10.937+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:11.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:10.937+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:11.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:10.937+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:11.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:10.938+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:11.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:10.786+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:11.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:10.786+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:11.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:10.786+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:11.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:10.786+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:11.636 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:11.278+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:11.636 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:11.278+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:11.636 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:11.278+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:11.636 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:11.279+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:11.915 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:11.787+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:11.915 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:11.787+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:11.915 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:11.787+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:11.915 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:11.788+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:12.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:11.918+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:12.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:11.918+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:12.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:11.918+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:12.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:11.919+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:12.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:12.263+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:12.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:12.263+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:12.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:12.263+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:12.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:12.263+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:13.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:12.784+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:13.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:12.784+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:13.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:12.784+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:13.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:12.785+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:13.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:12.870+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:13.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:12.870+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:13.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:12.870+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:13.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:12.871+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:13.524 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:13.251+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:13.524 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:13.251+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:13.524 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:13.251+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:13.524 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:13.252+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:13.819 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:13.770+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:13.819 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:13.770+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:13.819 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:13.770+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:13.819 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:13.771+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:14.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:13.822+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:14.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:13.822+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:14.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:13.822+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:14.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:13.823+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:14.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:14.235+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:14.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:14.235+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:14.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:14.235+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:14.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:14.235+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:15.094 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:14.846+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:15.094 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:14.846+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:15.094 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:14.846+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:15.094 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:14.847+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:15.095 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:14.776+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:15.095 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:14.776+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:15.095 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:14.776+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:15.095 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:14.777+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:15.470 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:57:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:57:15.097+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1208 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:57:15.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:15.278+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:15.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:15.278+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:15.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:15.278+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:15.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:15.279+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:15.812+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:15.812+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:15.812+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:15.812+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:16.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:15.886+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:16.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:15.886+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:16.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:15.886+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:16.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:15.887+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:16.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:16.292+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:16.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:16.292+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:16.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:16.292+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:16.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:16.292+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:17.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:16.897+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:17.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:16.897+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:17.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:16.897+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:17.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:16.898+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:17.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:16.853+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:17.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:16.853+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:17.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:16.853+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:17.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:16.854+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:17.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:17.316+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:17.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:17.316+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:17.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:17.316+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:17.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:17.317+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:18.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:17.900+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:18.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:17.900+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:18.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:17.900+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:18.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:17.900+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:18.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:17.868+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:18.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:17.868+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:18.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:17.868+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:18.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:17.868+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:18.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:18.278+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:18.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:18.278+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:18.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:18.278+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:18.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:18.278+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:19.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:18.871+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:19.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:18.871+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:19.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:18.871+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:19.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:18.871+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:19.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:18.847+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:19.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:18.847+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:19.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:18.847+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:19.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:18.848+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:19.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:19.307+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:19.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:19.307+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:19.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:19.307+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:19.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:19.308+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:20.116 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:57:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:57:20.097+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1253 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:57:20.117 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:19.864+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:20.117 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:19.864+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:20.117 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:19.864+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:20.117 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:19.865+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:20.117 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:19.862+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:20.117 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:19.862+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:20.117 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:19.862+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:20.117 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:19.863+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:20.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:20.325+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:20.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:20.325+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:20.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:20.325+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:20.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:20.325+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:21.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:20.834+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:21.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:20.834+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:21.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:20.834+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:21.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:20.835+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:21.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:20.910+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:21.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:20.910+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:21.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:20.910+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:21.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:20.911+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:21.637 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:21.338+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:21.637 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:21.338+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:21.637 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:21.338+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:21.637 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:21.339+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:21.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:21.803+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:21.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:21.803+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:21.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:21.803+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:21.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:21.804+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:21.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:21.864+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:21.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:21.864+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:21.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:21.864+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:21.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:21.865+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:22.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:22.307+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:22.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:22.307+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:22.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:22.307+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:22.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:22.308+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:23.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T08:57:23.086+0000 7ff55334f700 0 monclient(hunting): authenticate timed out after 300 2024-01-23T08:57:23.084 INFO:teuthology.orchestra.run.smithi059.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-01-23T08:57:23.112 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:22.907+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:23.113 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:22.907+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:23.113 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:22.907+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:23.113 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:22.908+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:23.113 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:22.849+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:23.113 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:22.849+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:23.113 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:22.849+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:23.113 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:22.849+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:23.419 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:23.266+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:23.419 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:23.266+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:23.419 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:23.266+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:23.419 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:23.267+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:23.888 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-01-23T08:57:24.163 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T08:57:24.219 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:23.805+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:24.219 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:23.805+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:24.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:23.805+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:24.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:23.805+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:24.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:23.906+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:24.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:23.906+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:24.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:23.906+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:24.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:23.907+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:24.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:24.221+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:24.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:24.221+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:24.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:24.221+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:24.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:24.222+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:25.095 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:24.763+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:25.096 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:24.763+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:25.096 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:24.763+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:25.096 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:24.764+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:25.096 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:24.923+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:25.096 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:24.923+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:25.096 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:24.923+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:25.096 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:24.924+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:25.429 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:57:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:57:25.097+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1301 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:57:25.429 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:25.216+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:25.429 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:25.216+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:25.429 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:25.216+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:25.429 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:25.217+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:25.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:25.876+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:25.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:25.876+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:25.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:25.876+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:25.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:25.877+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:25.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:25.720+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:25.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:25.720+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:25.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:25.720+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:25.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:25.721+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:26.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:26.197+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:26.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:26.197+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:26.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:26.197+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:26.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:26.198+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:26.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:26.834+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:26.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:26.834+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:26.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:26.834+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:26.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:26.835+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:26.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:26.693+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:26.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:26.693+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:26.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:26.693+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:26.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:26.694+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:27.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:27.218+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:27.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:27.218+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:27.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:27.218+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:27.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:27.219+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:27.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:27.874+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:27.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:27.874+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:27.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:27.874+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:27.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:27.875+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:27.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:27.688+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:27.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:27.688+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:27.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:27.688+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:27.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:27.689+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:28.644 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:28.233+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:28.644 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:28.233+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:28.644 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:28.233+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:28.644 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:28.234+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:28.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:28.646+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:28.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:28.646+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:28.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:28.646+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:28.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:28.647+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:29.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:28.922+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:29.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:28.922+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:29.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:28.922+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:29.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:28.923+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:29.631 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:29.252+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:29.631 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:29.252+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:29.631 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:29.252+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:29.631 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:29.253+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:29.918 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:29.633+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:29.918 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:29.633+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:29.919 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:29.633+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:29.919 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:29.634+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:30.217 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:57:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:57:30.098+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1346 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:57:30.217 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:29.920+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:30.217 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:29.920+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:30.217 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:29.920+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:30.217 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:29.921+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:30.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:30.256+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:30.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:30.256+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:30.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:30.256+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:30.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:30.256+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:30.880 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:30.617+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:30.880 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:30.617+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:30.880 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:30.617+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:30.880 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:30.618+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:31.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:30.882+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:31.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:30.882+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:31.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:30.882+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:31.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:30.883+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:31.638 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:31.283+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:31.638 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:31.283+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:31.638 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:31.283+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:31.638 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:31.284+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:31.918 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:31.654+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:31.918 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:31.654+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:31.918 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:31.654+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:31.918 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:31.654+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:32.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:31.920+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:32.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:31.920+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:32.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:31.920+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:32.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:31.921+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:32.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:32.285+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:32.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:32.285+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:32.669 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:32.285+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:32.669 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:32.286+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:32.970 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:32.914+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:32.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:32.914+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:32.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:32.914+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:32.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:32.915+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:32.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:32.670+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:32.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:32.670+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:32.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:32.670+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:32.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:32.671+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:33.524 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:33.248+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:33.525 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:33.248+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:33.525 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:33.248+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:33.525 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:33.248+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:33.877 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:33.640+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:33.878 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:33.640+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:33.878 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:33.640+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:33.878 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:33.640+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:34.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:33.880+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:34.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:33.880+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:34.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:33.880+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:34.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:33.881+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:34.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:34.293+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:34.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:34.293+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:34.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:34.293+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:34.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:34.294+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:34.970 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:34.881+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:34.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:34.881+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:34.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:34.881+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:34.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:34.882+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:34.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:34.669+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:34.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:34.669+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:34.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:34.669+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:34.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:34.670+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:35.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:57:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:57:35.098+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1393 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:57:35.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:35.292+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:35.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:35.292+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:35.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:35.292+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:35.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:35.292+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:35.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:35.705+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:35.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:35.705+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:35.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:35.705+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:35.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:35.706+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:35.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:35.920+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:35.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:35.920+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:35.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:35.920+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:35.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:35.921+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:36.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:36.301+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:36.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:36.301+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:36.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:36.301+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:36.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:36.302+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:37.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:36.729+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:37.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:36.729+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:37.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:36.729+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:37.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:36.730+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:37.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:36.962+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:37.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:36.962+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:37.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:36.962+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:37.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:36.963+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:37.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:37.287+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:37.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:37.287+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:37.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:37.287+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:37.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:37.288+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:38.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:37.942+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:38.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:37.942+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:38.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:37.942+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:38.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:37.943+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:38.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:37.778+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:38.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:37.778+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:38.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:37.778+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:38.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:37.779+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:38.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:38.323+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:38.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:38.323+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:38.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:38.323+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:38.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:38.324+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:39.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:38.801+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:39.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:38.801+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:39.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:38.801+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:39.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:38.802+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:39.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:38.984+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:39.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:38.984+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:39.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:38.984+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:39.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:38.985+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:39.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:39.291+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:39.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:39.291+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:39.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:39.291+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:39.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:39.291+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:40.097 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:40.027+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:40.097 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:40.027+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:40.097 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:40.027+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:40.097 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:40.028+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:40.098 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:39.818+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:40.098 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:39.818+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:40.098 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:39.818+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:40.098 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:39.819+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:40.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:57:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:57:40.100+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1497 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:57:40.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:40.329+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:40.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:40.329+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:40.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:40.329+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:40.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:40.330+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:41.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:41.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:41.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:41.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:41.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:41.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:41.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:41.054+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:41.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:40.869+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:41.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:40.869+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:41.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:40.869+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:41.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:40.869+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:41.639 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:41.320+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:41.639 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:41.320+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:41.639 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:41.320+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:41.639 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:41.320+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:41.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:41.832+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:41.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:41.832+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:41.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:41.832+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:41.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:41.833+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:42.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:42.031+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:42.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:42.031+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:42.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:42.031+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:42.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:42.032+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:42.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:42.273+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:42.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:42.273+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:42.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:42.273+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:42.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:42.274+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:43.066 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:42.818+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:43.066 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:42.818+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:43.067 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:42.818+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:43.067 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:42.818+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:43.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:43.069+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:43.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:43.069+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:43.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:43.069+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:43.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:43.070+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:43.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:43.235+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:43.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:43.235+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:43.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:43.235+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:43.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:43.236+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:44.203 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:44.034+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:44.203 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:44.034+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:44.203 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:44.034+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:44.203 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:44.034+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:44.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:43.849+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:44.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:43.849+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:44.204 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:43.849+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:44.204 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:43.851+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:44.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:44.206+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:44.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:44.206+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:44.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:44.206+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:44.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:44.206+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:45.173 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:57:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:57:45.100+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1594 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:57:45.173 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:45.002+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:45.173 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:45.002+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:45.173 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:45.002+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:45.173 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:45.003+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:45.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:44.888+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:45.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:44.888+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:45.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:44.888+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:45.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:44.889+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:45.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:45.176+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:45.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:45.176+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:45.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:45.176+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:45.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:45.177+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:46.173 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:45.991+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:46.173 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:45.991+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:46.173 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:45.991+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:46.173 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:45.993+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:46.173 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:45.892+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:46.173 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:45.892+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:46.173 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:45.892+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:46.173 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:45.893+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:46.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:46.176+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:46.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:46.176+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:46.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:46.176+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:46.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:46.176+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:47.214 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:46.965+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:47.215 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:46.965+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:47.215 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:46.965+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:47.215 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:46.966+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:47.215 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:46.886+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:47.215 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:46.886+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:47.215 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:46.886+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:47.215 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:46.886+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:47.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:47.217+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:47.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:47.217+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:47.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:47.217+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:47.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:47.218+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:48.218 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:47.907+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:48.218 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:47.907+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:48.218 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:47.907+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:48.218 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:47.908+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:48.218 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:47.950+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:48.218 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:47.950+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:48.218 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:47.950+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:48.218 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:47.951+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:48.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:48.221+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:48.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:48.221+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:48.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:48.221+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:48.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:48.221+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:49.171 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:48.918+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:49.171 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:48.918+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:49.171 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:48.918+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:49.172 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:48.919+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:49.172 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:49.174+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:49.172 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:49.174+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:49.172 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:49.174+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:49.172 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:48.903+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:49.172 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:48.903+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:49.172 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:48.903+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:49.172 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:48.903+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:49.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:49.175+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:50.202 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:57:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:57:50.101+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1689 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:57:50.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:49.913+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:50.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:49.913+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:50.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:49.913+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:50.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:49.913+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:50.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:49.878+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:50.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:49.878+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:50.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:49.878+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:50.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:49.878+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:50.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:50.205+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:50.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:50.205+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:50.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:50.205+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:50.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:50.205+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:51.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:50.876+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:51.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:50.876+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:51.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:50.876+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:51.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:50.877+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:51.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:50.904+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:51.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:50.904+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:51.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:50.904+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:51.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:50.904+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:51.640 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:51.235+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:51.640 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:51.235+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:51.640 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:51.235+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:51.640 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:51.235+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:51.896 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:51.861+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:51.897 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:51.861+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:51.897 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:51.861+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:51.897 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:51.861+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:52.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:51.900+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:52.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:51.900+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:52.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:51.900+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:52.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:51.901+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:52.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:52.241+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:52.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:52.241+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:52.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:52.241+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:52.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:52.242+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:53.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:52.866+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:53.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:52.866+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:53.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:52.866+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:53.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:52.867+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:53.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:52.859+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:53.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:52.859+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:53.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:52.859+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:53.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:52.860+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:53.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:53.289+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:53.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:53.289+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:53.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:53.289+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:53.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:53.290+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:54.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:53.823+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:54.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:53.823+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:54.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:53.823+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:54.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:53.824+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:54.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:53.907+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:54.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:53.907+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:54.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:53.907+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:54.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:53.908+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:54.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:54.308+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:54.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:54.308+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:54.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:54.308+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:54.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:54.309+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:55.107 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:57:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:57:55.102+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1786 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:57:55.107 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:54.902+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:55.108 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:54.902+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:55.108 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:54.902+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:55.108 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:54.904+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:55.108 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:54.859+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:55.108 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:54.859+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:55.108 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:54.859+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:55.108 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:54.859+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:55.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:55.298+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:55.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:55.298+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:55.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:55.298+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:55.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:55.299+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:56.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:55.822+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:56.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:55.822+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:56.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:55.822+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:56.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:55.822+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:56.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:55.925+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:56.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:55.925+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:56.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:55.925+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:56.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:55.926+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:56.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:56.303+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:56.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:56.303+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:56.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:56.303+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:56.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:56.303+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:57.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:56.869+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:57.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:56.869+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:57.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:56.869+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:57.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:56.870+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:57.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:56.898+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:57.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:56.898+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:57.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:56.898+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:57.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:56.898+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:57.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:57.258+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:57.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:57.258+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:57.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:57.258+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:57.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:57.259+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:58.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:57.898+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:58.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:57.898+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:58.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:57.898+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:58.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:57.898+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:58.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:57.941+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:58.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:57.941+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:58.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:57.941+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:58.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:57.942+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:58.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:58.229+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:58.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:58.229+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:58.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:58.229+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:58.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:58.230+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:59.192 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:58.972+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:59.192 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:58.972+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:59.192 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:58.972+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:57:59.192 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:58.973+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:57:59.193 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:58.904+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:59.193 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:58.904+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:59.193 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:58.904+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:57:59.193 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:58.905+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:57:59.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:59.195+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:59.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:59.195+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:59.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:59.195+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:57:59.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:57:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:57:59.196+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:00.173 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:58:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:58:00.102+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1880 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:58:00.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:59.942+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:00.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:59.942+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:00.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:59.942+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:00.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:57:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:57:59.943+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:00.174 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:59.920+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:00.174 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:59.920+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:00.175 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:59.920+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:00.175 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:57:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:57:59.921+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:00.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:00.177+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:00.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:00.177+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:00.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:00.177+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:00.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:00.177+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:01.166 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:00.970+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:01.166 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:00.970+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:01.166 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:00.970+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:01.166 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:00.970+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:01.166 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:00.914+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:01.167 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:00.914+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:01.167 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:00.914+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:01.167 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:00.915+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:01.167 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:01.169+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:01.167 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:01.169+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:01.167 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:01.169+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:01.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:01.169+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:02.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:01.947+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:02.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:01.947+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:02.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:01.947+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:02.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:01.948+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:02.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:01.941+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:02.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:01.941+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:02.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:01.941+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:02.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:01.942+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:02.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:02.187+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:02.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:02.187+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:02.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:02.187+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:02.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:02.188+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:03.204 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:02.958+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:03.204 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:02.958+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:03.204 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:02.958+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:03.204 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:02.959+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:03.205 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:02.943+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:03.205 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:02.943+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:03.205 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:02.943+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:03.205 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:02.944+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:03.205 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:03.207+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:03.205 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:03.207+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:03.205 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:03.207+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:03.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:03.208+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:04.220 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:04.192+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:04.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:04.192+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:04.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:04.192+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:04.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:04.193+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:04.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:03.951+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:04.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:03.951+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:04.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:03.951+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:04.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:03.952+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:04.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:03.977+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:04.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:03.977+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:04.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:03.977+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:04.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:03.978+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:05.220 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:58:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:58:05.103+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1978 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:58:05.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:04.959+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:05.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:04.959+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:05.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:04.959+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:05.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:04.960+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:05.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:05.186+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:05.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:05.186+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:05.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:05.186+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:05.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:05.187+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:05.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:04.945+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:05.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:04.945+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:05.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:04.945+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:05.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:04.946+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:06.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:05.991+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:06.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:05.991+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:06.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:05.991+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:06.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:05.992+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:06.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:05.963+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:06.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:05.963+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:06.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:05.963+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:06.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:05.964+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:06.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:06.234+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:06.234+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:06.234+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:06.235+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:07.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:07.011+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:07.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:07.011+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:07.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:07.011+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:07.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:07.011+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:07.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:06.961+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:07.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:06.961+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:07.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:06.961+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:07.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:06.962+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:07.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:07.249+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:07.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:07.249+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:07.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:07.249+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:07.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:07.250+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:08.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:08.055+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:08.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:08.055+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:08.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:08.055+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:08.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:08.056+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:08.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:07.952+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:08.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:07.952+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:08.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:07.952+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:08.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:07.952+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:08.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:08.246+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:08.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:08.246+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:08.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:08.246+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:08.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:08.247+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:09.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:08.959+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:09.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:08.959+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:09.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:08.959+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:09.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:08.960+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:09.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:09.101+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:09.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:09.101+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:09.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:09.101+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:09.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:09.102+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:09.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:09.244+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:09.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:09.244+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:09.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:09.244+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:09.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:09.245+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:10.470 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:58:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:58:10.103+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2073 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:58:10.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:09.989+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:10.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:09.989+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:10.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:09.989+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:10.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:09.990+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:10.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:10.081+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:10.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:10.081+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:10.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:10.081+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:10.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:10.081+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:10.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:10.239+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:10.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:10.239+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:10.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:10.239+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:10.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:10.239+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:11.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:11.093+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:11.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:11.093+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:11.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:11.093+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:11.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:11.094+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:11.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:11.204+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:11.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:11.204+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:11.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:11.204+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:11.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:11.205+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:11.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:10.955+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:11.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:10.955+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:11.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:10.955+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:11.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:10.956+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:12.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:12.074+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:12.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:12.074+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:12.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:12.074+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:12.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:12.075+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:12.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:12.166+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:12.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:12.166+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:12.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:12.166+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:12.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:12.166+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:12.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:11.959+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:12.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:11.959+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:12.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:11.959+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:12.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:11.960+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:13.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:12.929+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:13.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:12.929+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:13.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:12.929+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:13.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:12.929+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:13.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:13.044+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:13.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:13.044+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:13.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:13.044+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:13.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:13.046+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:13.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:13.203+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:13.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:13.203+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:13.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:13.203+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:13.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:13.203+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:14.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:14.091+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:14.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:14.091+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:14.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:14.091+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:14.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:14.092+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:14.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:13.939+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:14.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:13.939+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:14.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:13.939+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:14.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:13.940+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:14.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:14.244+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:14.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:14.244+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:14.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:14.244+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:14.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:14.245+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:15.220 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:58:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:58:15.104+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2172 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:58:15.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:15.079+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:15.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:15.079+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:15.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:15.079+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:15.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:15.079+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:15.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:14.935+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:15.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:14.935+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:15.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:14.935+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:15.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:14.936+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:15.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:15.267+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:15.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:15.267+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:15.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:15.267+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:15.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:15.268+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:16.045+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:16.045+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:16.045+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:16.045+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:16.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:15.887+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:16.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:15.887+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:16.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:15.887+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:16.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:15.887+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:16.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:16.229+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:16.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:16.229+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:16.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:16.229+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:16.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:16.230+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:17.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:17.029+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:17.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:17.029+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:17.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:17.029+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:17.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:17.030+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:17.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:16.910+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:17.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:16.911+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:17.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:16.911+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:17.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:16.911+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:17.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:17.228+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:17.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:17.228+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:17.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:17.228+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:17.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:17.229+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:18.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:17.998+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:18.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:17.998+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:18.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:17.998+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:18.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:17.998+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:18.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:17.899+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:18.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:17.899+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:18.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:17.899+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:18.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:17.900+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:18.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:18.224+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:18.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:18.224+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:18.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:18.224+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:18.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:18.225+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:19.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:19.000+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:19.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:19.000+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:19.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:19.000+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:19.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:19.001+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:19.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:18.892+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:19.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:18.892+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:19.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:18.892+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:19.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:18.893+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:19.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:19.263+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:19.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:19.263+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:19.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:19.263+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:19.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:19.264+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:20.102 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:19.850+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:20.102 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:19.850+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:20.102 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:19.850+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:20.102 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:19.851+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:20.102 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:20.022+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:20.102 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:20.022+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:20.102 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:20.022+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:20.103 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:20.022+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:20.470 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:58:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:58:20.105+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2264 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:58:20.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:20.303+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:20.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:20.303+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:20.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:20.303+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:20.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:20.303+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:21.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:21.036+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:21.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:21.036+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:21.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:21.036+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:21.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:21.037+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:21.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:20.869+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:21.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:20.869+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:21.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:20.869+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:21.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:20.870+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:21.641 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:21.324+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:21.641 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:21.324+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:21.641 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:21.324+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:21.641 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:21.324+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:22.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:21.911+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:22.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:21.911+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:22.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:21.911+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:22.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:21.911+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:22.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:22.012+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:22.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:22.012+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:22.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:22.012+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:22.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:22.013+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:22.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:22.332+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:22.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:22.332+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:22.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:22.332+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:22.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:22.333+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:23.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:23.001+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:23.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:23.001+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:23.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:23.001+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:23.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:23.002+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:23.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:22.959+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:23.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:22.959+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:23.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:22.959+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:23.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:22.960+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:23.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:23.320+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:23.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:23.320+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:23.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:23.320+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:23.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:23.320+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:24.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:23.981+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:24.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:23.981+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:24.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:23.981+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:24.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:23.982+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:24.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:23.960+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:24.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:23.960+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:24.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:23.960+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:24.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:23.961+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:24.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:24.300+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:24.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:24.300+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:24.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:24.300+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:24.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:24.300+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:25.220 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:58:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:58:25.105+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2361 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:58:25.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:24.990+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:25.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:24.990+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:25.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:24.990+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:25.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:24.990+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:25.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:24.940+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:25.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:24.940+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:25.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:24.941+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:25.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:24.941+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:25.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:25.296+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:25.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:25.296+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:25.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:25.296+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:25.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:25.297+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:26.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:26.038+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:26.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:26.038+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:26.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:26.038+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:26.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:26.039+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:26.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:25.936+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:26.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:25.936+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:26.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:25.937+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:26.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:25.937+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:26.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:26.333+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:26.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:26.333+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:26.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:26.333+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:26.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:26.333+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:27.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:27.006+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:27.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:27.006+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:27.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:27.006+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:27.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:27.006+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:27.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:26.952+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:27.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:26.952+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:27.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:26.952+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:27.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:26.952+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:27.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:27.333+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:27.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:27.333+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:27.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:27.333+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:27.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:27.333+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:28.366 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:27.974+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:28.366 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:27.974+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:28.366 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:27.974+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:28.366 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:27.975+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:28.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:27.997+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:28.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:27.997+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:28.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:27.997+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:28.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:27.998+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:28.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:28.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:28.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:28.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:28.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:28.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:28.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:28.369+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:29.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:28.968+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:29.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:28.968+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:29.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:28.968+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:29.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:28.969+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:29.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:29.003+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:29.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:29.003+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:29.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:29.003+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:29.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:29.004+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:29.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:29.343+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:29.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:29.343+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:29.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:29.343+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:29.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:29.344+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:30.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:58:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:58:30.106+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2454 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:58:30.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:29.930+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:30.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:29.930+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:30.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:29.930+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:30.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:29.931+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:30.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:30.011+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:30.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:30.011+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:30.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:30.011+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:30.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:30.012+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:30.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:30.354+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:30.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:30.354+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:30.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:30.354+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:30.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:30.355+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:31.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:31.014+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:31.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:31.014+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:31.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:31.014+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:31.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:31.015+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:31.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:30.950+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:31.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:30.950+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:31.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:30.950+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:31.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:30.950+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:31.642 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:31.347+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:31.642 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:31.347+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:31.642 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:31.347+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:31.642 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:31.347+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:32.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:32.015+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:32.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:32.015+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:32.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:32.015+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:32.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:32.016+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:32.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:31.907+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:32.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:31.907+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:32.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:31.907+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:32.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:31.908+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:32.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:32.317+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:32.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:32.317+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:32.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:32.317+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:32.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:32.317+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:33.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:33.047+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:33.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:33.047+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:33.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:33.047+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:33.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:33.048+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:33.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:32.908+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:33.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:32.908+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:33.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:32.908+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:33.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:32.910+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:33.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:33.339+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:33.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:33.340+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:33.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:33.340+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:33.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:33.340+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:34.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:34.085+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:34.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:34.085+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:34.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:34.085+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:34.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:34.086+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:34.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:33.876+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:34.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:33.876+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:34.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:33.876+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:34.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:33.877+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:34.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:34.341+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:34.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:34.341+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:34.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:34.341+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:34.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:34.342+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:35.092 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:35.094+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:35.092 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:35.095+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:35.092 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:35.095+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:35.093 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:35.095+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:35.093 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:34.845+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:35.093 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:34.845+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:35.093 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:34.845+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:35.093 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:34.846+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:35.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:58:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:58:35.107+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2551 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:58:35.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:35.305+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:35.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:35.305+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:35.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:35.305+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:35.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:35.305+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:36.098 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:35.823+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:36.098 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:35.823+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:36.098 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:35.823+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:36.098 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:35.824+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:36.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:36.101+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:36.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:36.101+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:36.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:36.101+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:36.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:36.102+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:36.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:36.347+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:36.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:36.347+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:36.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:36.347+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:36.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:36.347+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:37.058 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:36.791+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:37.059 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:36.791+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:37.059 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:36.791+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:37.059 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:36.792+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:37.323 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:37.061+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:37.323 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:37.061+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:37.323 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:37.061+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:37.323 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:37.062+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:37.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:37.326+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:37.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:37.326+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:37.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:37.326+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:37.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:37.326+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:38.107 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:37.828+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:38.108 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:37.828+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:38.108 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:37.828+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:38.108 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:37.829+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:38.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:38.344+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:38.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:38.345+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:38.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:38.345+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:38.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:38.345+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:38.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:38.110+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:38.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:38.110+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:38.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:38.110+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:38.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:38.111+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:39.129 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:38.786+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:39.129 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:38.786+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:39.129 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:38.786+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:39.129 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:38.787+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:39.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:39.319+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:39.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:39.319+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:39.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:39.319+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:39.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:39.319+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:39.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:39.132+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:39.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:39.132+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:39.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:39.132+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:39.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:39.133+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:40.084 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:39.831+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:40.085 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:39.831+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:40.085 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:39.831+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:40.085 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:39.831+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:40.470 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:58:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:58:40.106+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2644 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:58:40.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:40.277+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:40.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:40.277+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:40.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:40.277+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:40.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:40.277+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:40.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:40.086+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:40.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:40.086+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:40.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:40.086+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:40.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:40.087+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:41.125 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:40.836+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:41.125 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:40.836+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:41.125 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:40.836+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:41.125 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:40.837+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:41.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:41.272+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:41.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:41.272+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:41.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:41.272+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:41.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:41.273+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:41.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:41.127+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:41.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:41.127+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:41.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:41.127+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:41.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:41.128+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:41.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:41.830+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:41.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:41.830+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:41.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:41.830+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:41.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:41.831+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:42.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:42.113+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:42.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:42.113+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:42.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:42.113+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:42.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:42.113+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:42.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:42.279+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:42.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:42.279+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:42.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:42.279+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:42.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:42.279+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:43.064 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:42.796+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:43.064 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:42.796+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:43.065 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:42.796+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:43.065 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:42.797+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:43.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:43.066+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:43.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:43.066+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:43.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:43.066+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:43.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:43.067+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:43.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:43.293+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:43.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:43.293+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:43.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:43.293+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:43.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:43.294+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:44.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:43.841+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:44.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:43.841+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:44.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:43.841+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:44.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:43.842+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:44.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:44.045+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:44.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:44.045+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:44.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:44.045+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:44.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:44.046+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:44.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:44.298+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:44.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:44.298+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:44.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:44.298+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:44.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:44.299+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:45.078 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:44.795+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:45.078 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:44.795+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:45.078 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:44.795+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:45.078 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:44.796+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:45.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:58:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:58:45.107+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2741 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:58:45.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:45.080+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:45.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:45.080+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:45.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:45.080+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:45.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:45.080+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:45.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:45.270+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:45.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:45.270+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:45.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:45.270+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:45.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:45.271+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:46.040 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:45.762+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:46.040 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:45.762+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:46.040 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:45.763+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:46.040 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:45.763+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:46.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:46.042+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:46.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:46.042+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:46.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:46.042+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:46.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:46.043+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:46.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:46.268+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:46.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:46.268+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:46.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:46.268+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:46.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:46.268+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:47.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:46.999+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:47.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:46.999+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:47.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:46.999+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:47.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:47.000+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:47.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:46.757+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:47.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:46.757+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:47.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:46.757+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:47.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:46.757+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:47.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:47.303+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:47.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:47.303+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:47.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:47.303+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:47.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:47.303+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:48.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:47.780+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:48.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:47.780+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:48.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:47.780+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:48.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:47.781+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:48.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:47.984+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:48.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:47.984+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:48.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:47.984+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:48.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:47.985+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:48.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:48.337+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:48.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:48.337+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:48.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:48.337+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:48.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:48.338+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:49.012 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:48.746+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:49.012 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:48.746+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:49.012 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:48.746+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:49.013 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:48.747+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:49.374 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:49.014+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:49.375 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:49.014+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:49.375 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:49.014+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:49.375 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:49.015+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:49.715 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:49.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:49.716 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:49.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:49.716 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:49.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:49.716 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:49.377+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:49.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:49.717+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:49.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:49.717+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:49.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:49.717+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:49.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:49.718+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:50.417 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:58:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:58:50.108+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2834 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:58:50.417 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:50.008+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:50.417 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:50.008+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:50.417 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:50.008+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:50.417 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:50.009+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:50.709 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:50.419+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:50.709 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:50.419+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:50.709 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:50.419+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:50.709 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:50.419+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:50.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:50.711+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:50.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:50.711+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:50.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:50.711+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:50.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:50.712+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:51.385 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:50.998+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:51.385 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:50.998+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:51.385 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:50.998+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:51.385 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:50.999+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:51.643 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:51.387+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:51.643 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:51.387+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:51.643 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:51.387+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:51.643 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:51.387+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:51.967 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:51.685+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:51.967 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:51.685+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:51.967 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:51.685+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:51.967 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:51.686+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:52.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:51.969+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:52.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:51.969+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:52.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:51.969+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:52.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:51.970+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:52.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:52.423+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:52.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:52.423+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:52.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:52.423+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:52.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:52.424+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:52.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:52.661+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:52.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:52.661+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:52.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:52.661+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:52.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:52.661+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:53.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:53.016+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:53.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:53.016+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:53.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:53.016+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:53.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:53.016+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:53.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:53.469+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:53.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:53.469+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:53.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:53.469+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:53.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:53.470+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:53.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:53.625+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:53.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:53.625+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:53.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:53.625+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:53.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:53.626+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:54.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:54.029+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:54.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:54.029+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:54.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:54.029+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:54.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:54.029+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:54.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:54.473+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:54.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:54.473+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:54.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:54.473+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:54.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:54.474+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:54.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:54.595+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:54.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:54.595+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:54.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:54.595+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:54.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:54.596+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:55.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:58:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:58:55.108+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2924 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:58:55.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:55.038+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:55.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:55.038+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:55.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:55.038+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:55.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:55.040+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:55.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:55.637+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:55.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:55.637+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:55.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:55.637+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:55.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:55.638+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:55.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:55.493+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:55.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:55.493+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:55.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:55.493+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:55.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:55.494+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:56.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:56.072+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:56.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:56.072+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:56.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:56.072+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:56.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:56.073+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:56.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:56.507+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:56.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:56.507+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:56.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:56.507+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:56.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:56.508+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:56.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:56.665+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:56.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:56.665+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:56.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:56.665+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:56.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:56.666+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:57.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:57.064+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:57.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:57.064+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:57.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:57.064+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:57.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:57.064+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:57.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:57.625+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:57.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:57.625+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:57.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:57.625+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:57.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:57.625+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:57.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:57.484+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:57.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:57.484+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:57.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:57.484+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:57.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:57.484+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:58.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:58.035+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:58.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:58.035+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:58.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:58.035+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:58.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:58.036+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:58.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:58.600+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:58.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:58.600+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:58.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:58.600+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:58.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:58.600+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:58.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:58.447+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:58.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:58.447+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:58.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:58.447+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:58.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:58.448+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:59.421 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:59.050+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:59.421 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:59.050+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:59.421 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:59.050+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:58:59.422 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:58:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:58:59.050+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:58:59.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:59.563+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:59.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:59.563+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:59.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:59.563+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:58:59.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:58:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:58:59.563+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:58:59.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:59.423+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:59.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:59.423+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:59.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:59.423+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:58:59.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:58:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:58:59.424+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:00.416 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:59:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:59:00.110+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3018 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:59:00.416 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:00.066+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:00.417 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:00.066+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:00.417 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:00.066+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:00.417 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:00.067+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:00.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:00.531+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:00.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:00.531+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:00.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:00.531+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:00.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:00.532+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:00.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:00.419+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:00.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:00.419+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:00.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:00.419+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:00.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:00.420+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:01.424 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:01.101+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:01.424 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:01.101+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:01.424 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:01.101+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:01.425 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:01.102+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:01.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:01.575+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:01.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:01.575+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:01.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:01.575+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:01.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:01.576+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:01.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:01.427+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:01.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:01.427+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:01.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:01.427+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:01.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:01.428+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:02.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:02.110+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:02.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:02.110+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:02.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:02.110+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:02.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:02.111+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:02.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:02.577+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:02.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:02.577+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:02.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:02.577+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:02.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:02.578+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:02.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:02.470+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:02.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:02.470+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:02.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:02.470+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:02.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:02.471+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:03.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:03.158+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:03.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:03.158+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:03.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:03.158+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:03.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:03.159+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:03.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:03.613+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:03.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:03.613+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:03.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:03.613+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:03.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:03.614+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:03.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:03.499+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:03.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:03.499+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:03.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:03.499+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:03.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:03.500+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:04.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:04.120+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:04.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:04.120+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:04.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:04.120+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:04.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:04.121+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:04.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:04.453+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:04.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:04.453+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:04.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:04.453+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:04.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:04.454+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:04.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:04.657+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:04.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:04.657+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:04.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:04.657+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:04.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:04.658+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:05.470 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:59:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:59:05.111+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3117 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:59:05.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:05.107+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:05.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:05.107+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:05.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:05.107+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:05.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:05.108+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:05.859 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:05.678+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:05.859 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:05.679+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:05.859 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:05.679+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:05.859 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:05.679+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:05.859 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:05.487+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:05.859 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:05.487+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:05.859 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:05.487+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:05.859 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:05.488+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:06.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:06.111+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:06.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:06.111+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:06.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:06.111+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:06.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:06.111+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:06.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:06.681+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:06.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:06.681+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:06.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:06.681+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:06.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:06.682+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:06.445+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:06.445+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:06.445+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:06.446+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:07.419 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:07.120+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:07.419 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:07.120+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:07.419 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:07.120+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:07.419 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:07.121+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:07.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:07.422+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:07.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:07.422+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:07.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:07.422+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:07.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:07.423+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:08.077 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:07.727+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:08.077 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:07.727+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:08.077 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:07.727+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:08.078 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:07.728+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:08.430 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:08.080+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:08.430 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:08.080+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:08.430 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:08.080+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:08.430 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:08.081+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:08.684 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:08.433+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:08.684 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:08.433+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:08.684 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:08.433+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:08.684 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:08.434+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:08.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:08.687+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:08.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:08.687+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:08.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:08.687+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:08.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:08.688+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:09.385 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:09.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:09.386 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:09.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:09.386 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:09.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:09.386 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:09.054+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:09.709 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:09.389+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:09.710 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:09.389+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:09.710 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:09.389+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:09.710 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:09.389+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:09.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:09.713+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:09.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:09.713+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:09.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:09.713+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:09.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:09.714+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:10.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:59:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:59:10.112+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3212 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:59:10.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:10.048+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:10.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:10.048+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:10.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:10.048+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:10.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:10.048+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:10.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:10.341+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:10.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:10.341+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:10.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:10.341+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:10.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:10.342+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:11.022 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:10.751+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:11.022 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:10.751+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:11.022 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:10.751+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:11.022 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:10.752+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:11.384 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:11.025+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:11.384 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:11.025+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:11.384 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:11.025+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:11.384 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:11.026+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:11.644 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:11.387+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:11.644 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:11.387+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:11.644 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:11.387+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:11.644 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:11.388+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:11.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:11.785+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:11.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:11.785+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:11.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:11.785+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:11.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:11.786+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:12.367 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:12.008+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:12.367 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:12.008+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:12.367 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:12.008+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:12.367 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:12.009+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:12.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:12.370+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:12.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:12.370+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:12.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:12.370+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:12.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:12.371+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:13.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:12.804+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:13.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:12.804+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:13.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:12.804+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:13.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:12.805+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:13.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:13.045+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:13.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:13.045+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:13.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:13.045+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:13.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:13.046+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:13.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:13.332+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:13.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:13.332+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:13.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:13.332+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:13.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:13.333+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:14.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:13.833+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:14.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:13.833+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:14.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:13.833+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:14.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:13.834+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:14.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:14.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:14.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:14.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:14.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:14.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:14.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:14.053+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:14.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:14.359+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:14.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:14.359+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:14.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:14.359+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:14.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:14.360+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:15.121 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:59:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:59:15.113+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3311 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:59:15.121 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:14.867+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:15.121 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:14.867+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:15.121 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:14.867+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:15.121 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:14.868+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:15.122 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:15.009+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:15.122 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:15.009+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:15.122 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:15.009+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:15.122 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:15.009+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:15.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:15.333+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:15.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:15.333+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:15.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:15.333+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:15.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:15.334+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:15.985+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:15.985+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:15.985+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:15.986+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:16.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:15.879+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:16.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:15.879+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:16.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:15.879+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:16.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:15.880+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:16.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:16.314+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:16.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:16.314+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:16.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:16.314+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:16.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:16.314+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:17.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:16.968+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:17.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:16.968+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:17.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:16.968+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:17.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:16.969+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:17.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:16.857+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:17.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:16.857+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:17.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:16.857+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:17.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:16.858+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:17.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:17.326+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:17.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:17.326+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:17.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:17.326+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:17.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:17.327+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:18.148 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:17.844+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:18.148 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:17.844+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:18.148 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:17.844+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:18.148 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:17.845+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:18.148 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:17.965+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:18.148 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:17.965+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:18.148 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:17.965+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:18.149 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:17.966+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:18.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:18.370+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:18.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:18.370+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:18.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:18.370+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:18.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:18.370+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:19.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:18.976+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:19.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:18.976+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:19.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:18.976+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:19.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:18.977+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:19.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:18.810+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:19.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:18.810+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:19.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:18.810+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:19.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:18.811+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:19.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:19.367+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:19.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:19.367+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:19.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:19.367+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:19.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:19.368+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:20.110 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:19.792+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:20.111 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:19.792+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:20.111 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:19.792+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:20.111 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:19.793+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:20.111 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:20.020+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:20.111 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:20.020+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:20.111 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:20.020+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:20.111 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:20.021+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:20.395 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:59:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:59:20.113+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3405 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:59:20.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:20.398+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:20.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:20.398+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:20.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:20.398+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:20.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:20.399+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:21.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:20.976+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:21.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:20.976+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:21.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:20.976+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:21.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:20.977+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:21.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:20.828+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:21.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:20.828+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:21.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:20.828+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:21.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:20.829+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:21.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:21.432+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:21.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:21.432+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:21.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:21.432+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:21.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:21.433+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:22.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:22.015+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:22.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:22.015+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:22.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:22.015+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:22.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:22.015+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:22.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:21.858+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:22.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:21.858+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:22.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:21.858+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:22.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:21.859+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:22.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:22.428+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:22.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:22.428+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:22.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:22.428+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:22.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:22.429+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:23.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:22.987+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:23.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:22.987+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:23.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:22.987+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:23.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:22.987+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:23.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:22.833+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:23.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:22.833+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:23.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:22.833+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:23.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:22.834+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:23.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:23.442+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:23.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:23.442+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:23.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:23.442+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:23.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:23.442+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:24.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:23.969+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:24.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:23.969+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:24.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:23.969+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:24.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:23.969+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:24.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:23.816+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:24.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:23.816+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:24.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:23.816+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:24.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:23.816+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:24.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:24.485+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:24.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:24.485+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:24.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:24.485+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:24.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:24.485+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:25.111 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:24.851+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:25.111 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:24.851+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:25.111 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:24.851+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:25.111 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:24.852+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:25.112 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:24.928+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:25.112 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:24.928+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:25.112 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:24.928+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:25.112 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:24.929+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:25.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:59:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:59:25.114+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3503 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:59:25.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:25.452+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:25.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:25.452+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:25.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:25.452+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:25.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:25.453+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:26.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:25.903+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:26.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:25.903+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:26.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:25.903+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:26.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:25.903+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:26.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:25.825+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:26.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:25.825+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:26.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:25.825+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:26.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:25.826+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:26.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:26.431+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:26.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:26.431+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:26.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:26.431+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:26.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:26.432+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:27.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:26.886+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:27.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:26.886+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:27.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:26.886+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:27.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:26.887+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:27.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:26.855+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:27.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:26.855+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:27.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:26.855+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:27.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:26.856+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:27.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:27.399+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:27.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:27.399+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:27.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:27.399+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:27.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:27.400+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:28.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:27.896+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:28.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:27.896+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:28.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:27.896+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:28.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:27.897+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:28.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:27.831+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:28.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:27.831+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:28.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:27.831+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:28.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:27.832+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:28.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:28.422+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:28.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:28.422+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:28.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:28.422+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:28.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:28.422+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:29.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:28.945+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:29.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:28.945+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:29.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:28.945+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:29.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:28.945+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:29.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:28.805+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:29.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:28.805+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:29.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:28.805+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:29.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:28.806+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:29.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:29.467+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:29.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:29.467+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:29.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:29.467+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:29.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:29.467+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:30.112 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:29.923+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:30.112 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:29.923+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:30.112 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:29.923+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:30.112 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:29.923+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:30.112 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:29.808+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:30.112 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:29.808+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:30.112 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:29.808+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:30.113 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:29.809+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:30.452 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:59:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:59:30.115+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3596 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:59:30.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:30.455+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:30.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:30.455+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:30.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:30.455+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:30.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:30.456+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:31.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:30.855+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:31.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:30.855+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:31.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:30.855+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:31.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:30.856+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:31.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:30.898+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:31.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:30.898+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:31.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:30.898+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:31.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:30.898+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:31.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:31.458+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:31.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:31.458+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:31.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:31.458+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:31.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:31.458+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:32.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:31.892+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:32.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:31.892+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:32.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:31.892+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:32.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:31.893+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:32.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:31.886+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:32.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:31.886+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:32.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:31.886+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:32.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:31.887+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:32.891 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:32.505+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:32.891 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:32.505+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:32.891 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:32.505+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:32.891 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:32.505+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:33.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:32.935+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:33.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:32.935+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:33.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:32.935+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:33.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:32.936+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:33.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:32.894+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:33.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:32.894+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:33.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:32.894+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:33.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:32.895+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:33.915 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:33.542+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:33.915 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:33.542+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:33.915 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:33.542+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:33.915 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:33.543+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:34.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:33.971+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:34.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:33.971+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:34.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:33.971+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:34.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:33.972+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:34.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:33.918+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:34.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:33.918+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:34.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:33.918+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:34.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:33.919+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:34.917 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:34.518+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:34.917 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:34.518+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:34.917 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:34.518+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:34.917 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:34.518+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:35.220 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:59:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:59:35.115+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3694 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:59:35.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:34.920+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:35.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:34.920+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:35.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:34.920+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:35.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:34.921+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:35.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:35.009+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:35.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:35.009+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:35.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:35.009+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:35.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:35.010+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:35.909 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:35.489+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:35.910 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:35.489+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:35.910 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:35.489+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:35.910 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:35.489+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:36.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:35.981+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:36.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:35.981+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:36.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:35.981+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:36.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:35.982+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:36.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:35.913+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:36.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:35.913+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:36.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:35.913+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:36.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:35.914+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:36.925 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:36.500+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:36.925 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:36.500+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:36.925 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:36.500+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:36.925 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:36.501+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:37.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:36.928+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:37.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:36.928+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:37.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:36.928+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:37.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:36.929+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:37.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:36.933+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:37.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:36.933+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:37.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:36.933+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:37.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:36.934+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:37.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:37.468+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:37.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:37.468+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:37.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:37.468+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:37.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:37.469+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:38.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:37.918+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:38.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:37.918+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:38.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:37.918+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:38.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:37.919+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:38.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:37.910+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:38.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:37.910+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:38.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:37.910+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:38.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:37.911+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:38.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:38.449+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:38.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:38.449+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:38.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:38.449+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:38.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:38.450+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:39.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:38.954+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:39.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:38.954+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:39.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:38.954+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:39.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:38.955+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:39.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:38.955+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:39.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:38.955+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:39.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:38.955+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:39.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:38.956+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:39.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:39.409+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:39.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:39.409+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:39.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:39.409+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:39.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:39.410+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:40.220 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:59:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:59:40.116+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3788 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:59:40.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:39.910+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:40.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:39.910+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:40.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:39.910+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:40.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:39.911+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:40.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:39.946+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:40.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:39.946+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:40.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:39.946+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:40.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:39.946+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:40.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:40.444+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:40.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:40.444+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:40.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:40.444+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:40.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:40.445+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:41.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:40.908+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:41.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:40.908+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:41.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:40.908+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:41.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:40.909+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:41.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:40.896+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:41.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:40.896+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:41.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:40.896+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:41.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:40.897+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:41.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:41.441+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:41.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:41.441+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:41.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:41.441+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:41.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:41.442+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:42.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:41.885+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:42.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:41.885+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:42.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:41.885+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:42.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:41.886+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:42.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:41.936+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:42.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:41.936+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:42.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:41.936+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:42.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:41.937+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:42.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:42.429+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:42.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:42.429+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:42.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:42.429+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:42.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:42.430+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:43.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:42.859+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:43.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:42.859+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:43.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:42.859+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:43.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:42.860+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:43.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:42.927+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:43.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:42.927+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:43.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:42.927+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:43.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:42.928+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:43.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:43.473+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:43.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:43.473+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:43.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:43.473+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:43.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:43.474+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:44.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:43.882+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:44.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:43.882+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:44.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:43.882+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:44.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:43.883+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:44.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:43.821+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:44.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:43.821+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:44.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:43.821+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:44.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:43.822+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:44.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:44.457+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:44.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:44.457+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:44.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:44.457+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:44.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:44.458+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:45.114 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:44.886+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:45.114 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:44.886+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:45.114 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:44.886+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:45.114 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:44.886+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:45.115 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:44.801+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:45.115 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:44.801+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:45.115 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:44.801+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:45.115 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:44.801+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:45.435 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:59:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:59:45.117+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3886 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:59:45.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:45.438+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:45.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:45.438+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:45.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:45.438+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:45.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:45.439+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:46.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:45.755+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:46.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:45.755+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:46.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:45.755+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:46.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:45.756+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:46.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:45.893+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:46.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:45.893+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:46.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:45.893+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:46.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:45.894+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:46.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:46.472+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:46.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:46.472+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:46.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:46.472+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:46.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:46.473+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:47.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:46.784+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:47.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:46.784+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:47.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:46.784+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:47.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:46.784+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:47.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:46.911+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:47.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:46.911+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:47.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:46.911+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:47.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:46.912+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:47.793 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:47.494+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:47.794 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:47.494+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:47.794 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:47.494+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:47.794 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:47.495+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:48.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:47.797+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:48.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:47.797+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:48.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:47.797+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:48.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:47.798+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:48.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:47.930+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:48.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:47.930+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:48.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:47.930+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:48.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:47.931+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:48.842 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:48.494+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:48.842 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:48.494+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:48.843 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:48.494+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:48.843 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:48.494+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:49.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:48.910+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:49.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:48.910+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:49.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:48.910+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:49.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:48.911+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:49.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:48.845+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:49.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:48.845+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:49.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:48.845+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:49.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:48.846+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:49.876 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:49.535+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:49.877 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:49.535+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:49.877 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:49.535+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:49.877 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:49.536+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:50.137 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:59:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:59:50.118+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3980 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:59:50.137 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:49.879+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:50.137 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:49.879+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:50.137 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:49.880+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:50.137 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:49.880+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:50.137 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:49.881+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:50.138 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:49.881+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:50.138 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:49.881+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:50.138 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:49.881+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:50.842 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:50.550+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:50.843 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:50.550+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:50.843 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:50.550+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:50.843 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:50.551+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:51.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:50.846+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:51.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:50.846+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:51.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:50.846+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:51.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:50.847+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:51.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:50.895+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:51.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:50.895+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:51.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:50.895+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:51.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:50.896+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:51.848 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:51.565+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:51.848 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:51.565+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:51.848 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:51.565+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:51.848 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:51.565+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:52.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:51.851+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:52.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:51.851+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:52.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:51.851+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:52.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:51.852+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:52.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:51.881+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:52.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:51.881+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:52.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:51.881+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:52.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:51.882+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:52.820 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:52.545+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:52.820 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:52.545+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:52.820 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:52.545+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:52.820 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:52.546+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:53.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:52.823+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:53.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:52.823+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:53.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:52.823+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:53.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:52.824+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:53.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:52.846+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:53.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:52.846+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:53.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:52.846+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:53.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:52.847+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:53.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:53.560+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:53.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:53.560+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:53.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:53.560+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:53.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:53.561+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:54.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:53.814+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:54.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:53.814+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:54.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:53.814+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:54.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:53.815+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:54.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:53.876+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:54.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:53.876+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:54.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:53.876+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:54.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:53.877+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:54.823 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:54.539+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:54.823 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:54.539+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:54.823 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:54.539+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:54.823 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:54.540+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:55.115 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:54.873+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:55.116 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:54.873+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:55.116 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:54.873+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:55.116 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:54.874+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:55.116 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:54.826+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:55.116 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:54.826+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:55.116 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:54.826+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:55.116 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:54.827+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:55.470 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 08:59:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T08:59:55.119+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4077 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T08:59:55.848 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:55.498+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:55.848 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:55.498+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:55.849 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:55.498+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:55.849 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:55.499+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:56.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:55.851+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:56.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:55.851+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:56.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:55.851+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:56.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:55.852+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:56.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:55.901+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:56.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:55.901+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:56.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:55.901+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:56.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:55.902+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:56.813 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:56.542+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:56.813 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:56.542+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:56.813 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:56.542+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:56.813 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:56.543+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:57.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:56.901+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:57.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:56.901+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:57.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:56.901+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:57.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:56.902+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:57.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:56.816+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:57.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:56.816+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:57.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:56.816+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:57.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:56.817+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:57.843 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:57.570+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:57.843 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:57.570+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:57.843 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:57.570+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:57.843 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:57.571+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:58.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:57.846+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:58.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:57.846+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:58.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:57.846+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:58.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:57.847+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:58.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:57.881+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:58.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:57.881+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:58.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:57.881+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:58.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:57.881+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:58.813 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:58.523+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:58.813 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:58.523+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:58.813 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:58.523+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:58.813 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:58.524+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:59.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:58.816+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:59.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:58.816+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:59.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:58.816+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T08:59:59.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:58.817+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T08:59:59.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:58.843+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:59.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:58.843+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:59.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:58.843+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T08:59:59.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:58.844+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T08:59:59.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:59.477+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:59.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:59.477+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:59.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:59.477+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T08:59:59.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 08:59:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T08:59:59.478+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:00.116 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:59.842+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:00.117 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:59.842+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:00.117 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:59.842+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:00.117 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 08:59:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T08:59:59.843+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:00.117 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:59.800+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:00.117 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:59.800+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:00.117 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:59.800+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:00.118 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 08:59:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T08:59:59.800+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:00.444 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:00:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:00:00.119+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4171 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:00:00.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:00.446+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:00.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:00.446+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:00.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:00.446+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:00.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:00.447+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:01.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:00.844+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:01.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:00.844+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:01.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:00.844+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:01.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:00.845+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:01.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:00.813+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:01.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:00.813+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:01.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:00.813+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:01.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:00.814+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:01.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:01.453+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:01.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:01.453+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:01.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:01.453+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:01.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:01.454+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:02.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:01.886+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:02.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:01.886+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:02.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:01.886+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:02.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:01.887+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:02.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:01.835+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:02.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:01.835+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:02.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:01.835+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:02.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:01.836+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:02.801 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:02.473+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:02.801 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:02.473+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:02.801 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:02.473+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:02.801 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:02.474+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:03.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:02.803+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:03.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:02.803+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:03.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:02.803+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:03.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:02.804+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:03.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:02.866+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:03.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:02.866+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:03.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:02.866+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:03.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:02.867+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:03.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:03.462+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:03.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:03.462+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:03.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:03.462+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:03.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:03.463+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:04.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:03.900+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:04.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:03.900+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:04.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:03.900+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:04.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:03.901+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:04.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:03.803+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:04.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:03.803+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:04.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:03.804+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:04.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:03.804+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:04.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:04.459+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:04.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:04.459+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:04.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:04.459+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:04.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:04.459+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:05.117 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:04.868+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:05.117 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:04.868+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:05.117 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:04.868+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:05.117 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:04.868+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:05.118 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:04.815+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:05.118 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:04.815+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:05.118 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:04.815+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:05.118 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:04.815+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:05.462 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:00:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:00:05.119+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4269 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:00:05.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:05.464+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:05.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:05.464+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:05.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:05.464+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:05.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:05.465+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:06.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:05.825+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:06.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:05.825+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:06.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:05.825+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:06.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:05.826+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:06.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:05.855+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:06.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:05.855+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:06.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:05.855+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:06.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:05.856+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:06.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:06.425+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:06.426+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:06.426+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:06.427+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:07.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:06.813+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:07.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:06.813+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:07.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:06.813+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:07.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:06.813+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:07.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:06.899+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:07.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:06.899+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:07.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:06.899+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:07.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:06.900+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:07.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:07.394+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:07.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:07.394+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:07.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:07.394+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:07.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:07.395+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:08.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:07.924+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:08.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:07.924+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:08.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:07.924+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:08.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:07.925+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:08.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:07.862+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:08.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:07.862+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:08.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:07.862+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:08.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:07.863+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:08.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:08.405+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:08.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:08.405+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:08.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:08.406+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:08.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:08.406+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:09.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:08.825+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:09.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:08.825+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:09.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:08.825+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:09.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:08.826+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:09.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:08.907+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:09.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:08.907+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:09.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:08.907+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:09.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:08.907+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:09.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:09.389+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:09.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:09.389+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:09.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:09.389+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:09.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:09.390+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:10.118 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:09.925+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:10.118 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:09.925+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:10.118 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:09.925+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:10.118 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:09.926+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:10.118 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:09.794+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:10.119 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:09.794+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:10.119 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:09.794+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:10.119 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:09.794+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:10.371 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:00:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:00:10.120+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4363 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:00:10.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:10.386+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:10.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:10.386+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:10.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:10.386+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:10.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:10.387+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:11.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:10.958+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:11.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:10.958+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:11.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:10.958+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:11.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:10.959+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:11.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:10.792+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:11.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:10.792+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:11.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:10.792+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:11.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:10.793+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:11.648 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:11.373+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:11.648 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:11.373+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:11.649 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:11.373+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:11.649 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:11.374+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:11.970 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:11.788+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:11.970 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:11.788+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:11.970 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:11.788+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:11.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:11.789+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:12.364 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:11.983+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:12.365 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:11.983+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:12.365 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:11.983+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:12.365 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:11.983+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:12.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:12.367+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:12.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:12.367+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:12.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:12.367+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:12.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:12.367+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:13.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:12.819+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:13.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:12.819+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:13.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:12.819+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:13.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:12.820+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:13.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:12.941+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:13.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:12.941+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:13.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:12.941+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:13.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:12.942+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:13.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:13.408+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:13.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:13.408+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:13.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:13.408+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:13.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:13.408+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:14.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:13.779+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:14.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:13.779+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:14.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:13.779+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:14.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:13.780+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:14.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:13.926+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:14.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:13.926+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:14.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:13.926+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:14.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:13.927+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:14.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:14.389+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:14.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:14.389+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:14.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:14.389+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:14.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:14.389+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:15.118 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:14.932+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:15.119 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:14.932+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:15.119 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:14.932+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:15.119 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:14.933+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:15.119 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:14.813+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:15.119 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:14.813+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:15.119 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:14.813+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:15.119 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:14.814+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:15.470 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:00:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:00:15.120+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4462 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:00:15.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:15.367+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:15.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:15.367+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:15.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:15.367+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:15.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:15.367+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:16.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:15.795+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:16.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:15.795+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:15.795+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:15.796+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:16.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:15.977+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:16.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:15.977+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:16.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:15.977+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:16.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:15.978+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:16.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:16.354+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:16.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:16.354+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:16.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:16.354+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:16.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:16.355+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:17.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:16.846+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:17.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:16.846+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:17.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:16.846+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:17.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:16.847+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:17.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:16.978+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:17.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:16.978+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:17.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:16.978+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:17.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:16.979+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:17.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:17.373+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:17.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:17.373+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:17.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:17.373+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:17.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:17.373+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:18.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:17.945+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:18.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:17.945+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:18.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:17.945+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:18.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:17.946+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:18.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:17.880+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:18.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:17.880+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:18.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:17.880+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:18.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:17.881+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:18.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:18.386+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:18.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:18.386+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:18.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:18.386+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:18.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:18.386+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:19.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:18.943+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:19.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:18.943+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:19.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:18.943+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:19.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:18.944+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:19.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:18.859+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:19.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:18.859+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:19.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:18.859+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:19.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:18.860+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:19.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:19.378+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:19.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:19.378+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:19.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:19.378+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:19.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:19.379+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:20.120 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:19.991+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:20.120 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:19.991+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:20.120 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:19.991+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:20.120 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:19.992+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:20.120 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:19.871+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:20.120 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:19.871+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:20.120 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:19.871+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:20.120 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:19.872+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:20.401 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:00:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:00:20.122+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4556 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:00:20.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:20.404+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:20.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:20.404+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:20.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:20.404+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:20.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:20.405+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:21.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:20.982+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:21.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:20.982+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:21.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:20.982+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:21.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:20.983+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:21.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:20.882+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:21.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:20.882+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:21.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:20.882+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:21.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:20.883+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:21.649 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:21.357+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:21.649 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:21.357+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:21.649 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:21.357+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:21.649 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:21.358+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:22.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:21.912+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:22.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:21.912+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:22.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:21.912+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:22.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:21.912+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:22.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:21.962+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:22.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:21.962+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:22.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:21.962+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:22.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:21.963+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:22.659 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:22.323+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:22.659 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:22.323+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:22.659 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:22.323+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:22.659 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:22.324+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:22.940 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:22.899+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:22.940 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:22.899+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:22.940 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:22.899+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:22.940 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:22.900+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:23.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:22.943+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:23.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:22.943+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:23.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:22.943+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:23.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:22.944+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:23.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:23.358+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:23.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:23.358+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:23.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:23.358+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:23.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:23.358+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:24.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:23.892+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:24.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:23.892+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:24.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:23.892+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:24.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:23.892+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:24.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:23.984+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:24.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:23.984+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:24.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:23.984+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:24.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:23.985+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:24.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:24.380+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:24.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:24.380+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:24.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:24.380+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:24.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:24.380+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:25.220 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:00:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:00:25.123+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4658 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:00:25.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:24.949+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:25.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:24.949+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:25.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:24.949+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:25.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:24.950+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:25.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:24.909+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:25.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:24.909+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:25.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:24.909+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:25.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:24.910+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:25.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:25.403+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:25.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:25.403+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:25.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:25.403+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:25.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:25.404+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:26.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:25.970+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:26.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:25.970+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:26.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:25.970+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:26.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:25.971+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:26.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:25.870+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:26.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:25.870+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:26.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:25.870+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:26.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:25.871+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:26.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:26.441+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:26.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:26.441+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:26.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:26.441+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:26.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:26.441+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:27.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:26.928+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:27.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:26.928+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:27.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:26.928+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:27.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:26.929+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:27.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:26.906+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:27.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:26.906+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:27.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:26.906+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:27.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:26.907+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:27.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:27.427+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:27.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:27.427+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:27.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:27.427+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:27.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:27.427+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:28.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:27.964+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:28.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:27.964+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:28.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:27.964+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:28.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:27.965+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:28.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:27.941+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:28.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:27.941+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:28.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:27.941+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:28.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:27.942+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:28.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:28.473+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:28.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:28.473+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:28.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:28.473+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:28.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:28.474+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:29.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:28.968+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:29.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:28.968+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:29.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:28.968+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:29.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:28.969+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:29.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:28.934+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:29.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:28.934+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:29.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:28.934+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:29.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:28.934+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:29.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:29.442+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:29.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:29.442+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:29.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:29.442+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:29.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:29.443+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:30.220 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:00:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:00:30.124+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4761 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:00:30.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:29.926+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:30.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:29.926+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:30.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:29.926+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:30.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:29.927+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:30.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:29.990+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:30.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:29.990+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:30.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:29.990+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:30.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:29.990+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:30.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:30.423+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:30.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:30.423+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:30.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:30.423+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:30.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:30.424+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:31.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:31.035+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:31.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:31.035+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:31.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:31.035+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:31.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:31.035+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:31.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:30.908+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:31.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:30.908+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:31.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:30.908+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:31.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:30.909+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:31.649 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:31.380+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:31.650 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:31.380+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:31.650 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:31.380+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:31.650 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:31.381+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:31.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:31.890+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:31.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:31.890+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:31.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:31.890+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:31.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:31.891+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:32.423 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:31.996+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:32.423 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:31.996+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:32.423 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:31.996+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:32.423 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:31.997+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:32.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:32.426+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:32.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:32.426+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:32.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:32.426+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:32.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:32.427+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:33.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:32.971+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:33.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:32.971+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:33.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:32.971+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:33.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:32.972+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:33.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:32.906+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:33.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:32.906+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:33.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:32.906+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:33.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:32.907+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:33.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:33.426+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:33.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:33.426+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:33.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:33.426+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:33.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:33.427+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:34.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:33.935+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:34.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:33.935+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:34.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:33.935+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:34.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:33.936+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:34.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:33.894+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:34.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:33.894+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:34.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:33.894+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:34.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:33.895+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:34.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:34.448+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:34.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:34.448+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:34.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:34.448+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:34.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:34.448+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:35.220 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:00:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:00:35.124+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4864 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:00:35.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:34.946+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:35.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:34.946+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:35.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:34.946+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:35.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:34.947+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:35.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:34.915+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:35.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:34.915+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:35.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:34.915+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:35.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:34.917+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:35.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:35.403+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:35.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:35.403+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:35.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:35.403+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:35.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:35.403+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:36.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:35.944+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:36.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:35.944+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:36.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:35.944+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:36.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:35.944+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:36.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:35.957+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:36.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:35.957+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:36.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:35.957+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:36.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:35.958+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:36.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:36.421+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:36.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:36.421+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:36.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:36.421+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:36.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:36.422+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:37.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:36.948+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:37.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:36.948+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:37.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:36.948+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:37.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:36.948+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:37.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:36.943+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:37.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:36.943+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:37.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:36.943+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:37.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:36.943+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:37.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:37.375+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:37.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:37.375+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:37.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:37.375+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:37.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:37.375+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:38.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:37.993+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:38.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:37.993+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:38.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:37.993+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:38.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:37.994+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:38.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:37.957+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:38.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:37.957+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:38.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:37.957+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:38.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:37.958+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:38.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:38.344+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:38.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:38.344+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:38.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:38.344+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:38.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:38.345+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:39.305 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:39.000+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:39.305 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:39.000+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:39.306 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:39.000+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:39.306 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:39.002+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:39.306 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:39.009+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:39.306 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:39.009+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:39.306 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:39.009+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:39.306 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:39.010+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:39.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:39.308+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:39.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:39.308+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:39.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:39.308+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:39.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:39.309+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:40.245 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:00:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:00:40.125+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4959 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:00:40.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:39.977+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:40.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:39.977+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:40.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:39.977+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:40.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:39.978+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:40.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:40.031+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:40.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:40.031+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:40.247 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:40.031+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:40.247 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:40.031+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:40.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:40.295+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:40.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:40.295+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:40.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:40.295+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:40.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:40.296+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:41.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:41.004+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:41.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:41.004+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:41.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:41.004+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:41.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:41.005+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:41.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:40.946+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:41.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:40.947+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:41.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:40.947+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:41.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:40.948+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:41.650 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:41.269+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:41.650 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:41.269+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:41.650 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:41.269+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:41.650 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:41.270+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:42.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:42.041+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:42.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:42.041+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:42.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:42.041+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:42.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:42.042+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:42.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:41.921+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:42.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:41.921+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:42.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:41.921+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:42.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:41.922+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:42.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:42.249+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:42.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:42.249+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:42.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:42.249+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:42.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:42.250+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:43.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:43.019+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:43.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:43.019+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:43.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:43.019+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:43.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:43.020+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:43.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:42.956+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:43.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:42.956+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:43.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:42.956+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:43.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:42.957+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:43.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:43.295+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:43.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:43.295+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:43.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:43.295+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:43.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:43.296+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:44.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:44.065+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:44.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:44.065+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:44.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:44.065+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:44.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:44.065+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:44.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:43.961+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:44.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:43.961+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:44.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:43.961+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:44.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:43.962+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:44.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:44.251+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:44.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:44.251+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:44.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:44.251+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:44.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:44.252+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:45.245 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:00:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:00:45.126+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5065 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:00:45.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:45.052+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:45.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:45.052+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:45.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:45.052+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:45.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:45.053+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:45.245 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:44.990+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:45.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:44.990+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:45.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:44.990+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:45.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:44.991+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:45.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:45.276+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:45.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:45.276+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:45.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:45.276+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:45.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:45.276+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:46.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:46.086+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:46.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:46.086+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:46.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:46.086+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:46.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:46.087+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:46.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:45.953+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:46.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:45.953+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:46.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:45.953+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:46.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:45.954+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:46.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:46.306+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:46.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:46.306+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:46.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:46.306+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:46.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:46.306+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:47.349 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:47.049+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:47.349 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:47.049+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:47.349 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:47.049+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:47.349 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:47.049+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:47.350 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:46.985+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:47.350 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:46.985+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:47.350 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:46.985+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:47.350 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:46.985+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:47.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:47.352+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:47.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:47.352+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:47.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:47.352+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:47.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:47.353+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:48.362 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:48.000+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:48.362 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:48.000+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:48.362 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:48.000+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:48.363 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:48.001+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:48.363 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:48.025+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:48.363 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:48.025+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:48.363 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:48.025+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:48.363 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:48.025+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:48.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:48.365+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:48.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:48.365+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:48.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:48.365+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:48.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:48.366+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:49.406 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:49.024+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:49.406 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:49.024+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:49.407 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:49.024+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:49.407 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:49.025+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:49.407 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:49.029+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:49.407 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:49.029+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:49.407 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:49.029+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:49.407 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:49.030+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:49.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:49.409+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:49.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:49.409+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:49.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:49.409+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:49.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:49.410+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:50.410 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:00:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:00:50.127+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5160 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:00:50.411 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:50.048+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:50.411 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:50.048+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:50.411 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:50.048+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:50.411 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:50.049+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:50.411 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:50.007+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:50.411 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:50.007+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:50.412 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:50.007+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:50.412 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:50.008+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:50.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:50.414+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:50.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:50.414+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:50.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:50.414+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:50.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:50.415+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:51.373 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:51.040+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:51.373 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:51.040+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:51.373 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:51.040+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:51.373 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:51.041+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:51.373 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:51.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:51.374 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:51.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:51.374 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:51.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:51.374 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:51.053+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:51.374 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:51.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:51.374 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:51.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:51.374 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:51.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:51.650 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:51.377+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:52.361 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:52.074+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:52.361 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:52.074+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:52.361 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:52.074+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:52.361 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:52.075+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:52.361 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:52.023+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:52.361 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:52.023+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:52.361 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:52.023+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:52.361 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:52.023+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:52.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:52.364+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:52.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:52.364+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:52.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:52.364+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:52.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:52.365+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:53.316 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:53.037+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:53.316 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:53.037+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:53.316 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:53.037+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:53.316 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:53.038+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:53.316 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:53.103+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:53.316 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:53.103+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:53.316 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:53.103+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:53.316 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:53.104+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:53.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:53.319+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:53.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:53.319+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:53.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:53.319+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:53.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:53.320+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:54.365 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:54.119+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:54.365 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:54.119+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:54.365 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:54.119+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:54.366 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:54.120+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:54.366 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:54.029+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:54.366 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:54.029+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:54.366 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:54.029+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:54.366 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:54.030+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:54.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:54.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:54.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:54.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:54.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:54.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:54.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:54.369+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:55.348 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:00:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:00:55.128+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5256 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:00:55.348 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:55.050+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:55.348 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:55.050+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:55.348 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:55.050+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:55.348 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:55.051+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:55.349 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:55.105+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:55.349 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:55.105+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:55.349 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:55.105+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:55.349 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:55.106+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:55.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:55.351+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:55.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:55.351+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:55.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:55.351+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:55.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:55.352+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:56.344 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:56.092+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:56.344 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:56.092+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:56.344 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:56.092+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:56.344 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:56.093+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:56.344 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:56.084+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:56.344 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:56.084+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:56.344 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:56.084+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:56.344 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:56.085+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:56.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:56.347+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:56.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:56.347+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:56.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:56.347+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:56.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:56.348+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:57.382 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:57.125+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:57.383 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:57.125+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:57.383 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:57.125+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:57.383 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:57.126+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:57.383 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:57.048+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:57.383 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:57.048+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:57.383 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:57.048+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:57.383 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:57.048+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:57.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:57.386+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:57.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:57.386+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:57.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:57.386+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:57.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:57.386+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:58.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:58.120+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:58.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:58.120+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:58.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:58.120+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:58.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:58.120+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:58.433 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:58.062+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:58.433 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:58.062+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:58.433 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:58.062+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:58.433 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:58.062+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:58.433 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:58.435+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:58.433 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:58.435+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:58.433 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:58.435+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:58.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:58.436+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:59.418 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:59.064+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:59.419 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:59.064+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:59.419 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:59.064+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:00:59.419 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:00:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:00:59.065+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:00:59.419 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:59.077+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:59.419 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:59.077+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:59.419 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:59.077+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:00:59.419 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:00:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:00:59.078+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:00:59.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:59.422+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:59.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:59.422+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:59.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:59.422+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:00:59.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:00:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:00:59.422+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:00.427 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:01:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:01:00.129+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5355 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:01:00.427 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:00.066+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:00.427 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:00.066+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:00.427 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:00.066+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:00.427 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:00.067+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:00.428 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:00.067+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:00.428 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:00.067+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:00.428 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:00.067+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:00.428 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:00.068+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:00.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:00.430+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:00.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:00.430+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:00.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:00.430+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:00.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:00.431+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:01.404 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:01.084+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:01.405 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:01.084+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:01.405 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:01.084+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:01.405 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:01.085+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:01.405 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:01.043+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:01.405 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:01.043+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:01.405 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:01.043+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:01.405 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:01.044+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:01.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:01.407+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:01.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:01.407+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:01.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:01.407+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:01.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:01.408+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:02.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:02.031+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:02.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:02.031+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:02.448 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:02.031+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:02.448 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:02.032+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:02.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:02.120+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:02.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:02.120+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:02.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:02.120+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:02.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:02.121+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:02.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:02.450+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:02.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:02.450+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:02.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:02.450+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:02.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:02.451+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:03.460 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:03.092+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:03.460 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:03.092+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:03.460 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:03.092+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:03.460 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:03.093+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:03.460 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:02.993+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:03.460 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:02.993+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:03.460 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:02.993+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:03.460 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:02.994+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:03.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:03.463+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:03.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:03.463+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:03.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:03.463+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:03.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:03.464+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:04.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:03.974+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:04.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:03.974+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:04.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:03.974+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:04.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:03.974+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:04.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:04.133+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:04.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:04.133+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:04.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:04.133+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:04.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:04.134+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:04.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:04.472+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:04.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:04.472+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:04.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:04.472+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:04.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:04.472+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:05.220 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:01:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:01:05.129+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5474 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:01:05.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:05.166+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:05.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:05.166+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:05.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:05.166+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:05.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:05.167+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:05.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:04.949+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:05.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:04.949+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:05.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:04.949+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:05.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:04.949+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:05.667 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:05.475+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:05.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:05.475+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:05.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:05.475+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:05.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:05.476+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:06.461 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:05.978+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:06.461 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:05.978+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:06.461 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:05.978+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:06.461 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:05.979+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:06.461 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:06.166+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:06.462 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:06.166+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:06.462 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:06.166+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:06.462 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:06.167+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:06.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:06.464+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:06.464+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:06.464+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:06.465+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:07.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:06.958+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:07.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:06.958+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:07.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:06.958+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:07.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:06.959+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:07.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:07.163+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:07.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:07.163+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:07.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:07.163+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:07.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:07.164+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:07.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:07.468+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:07.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:07.468+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:07.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:07.468+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:07.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:07.468+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:08.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:07.910+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:08.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:07.910+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:08.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:07.910+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:08.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:07.911+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:08.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:08.116+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:08.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:08.116+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:08.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:08.116+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:08.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:08.117+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:08.739 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:08.477+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:08.739 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:08.477+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:08.739 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:08.477+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:08.740 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:08.478+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:09.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:08.914+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:09.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:08.914+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:09.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:08.914+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:09.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:08.915+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:09.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:09.072+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:09.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:09.072+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:09.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:09.072+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:09.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:09.073+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:09.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:09.455+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:09.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:09.455+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:09.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:09.455+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:09.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:09.456+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:10.127 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:09.870+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:10.127 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:09.870+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:10.128 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:09.870+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:10.128 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:09.871+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:10.128 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:10.116+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:10.128 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:10.116+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:10.128 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:10.116+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:10.128 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:10.117+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:10.428 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:01:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:01:10.130+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5582 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:01:10.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:10.432+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:10.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:10.432+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:10.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:10.432+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:10.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:10.433+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:11.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:10.913+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:11.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:10.913+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:11.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:10.913+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:11.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:10.914+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:11.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:11.077+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:11.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:11.077+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:11.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:11.077+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:11.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:11.078+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:11.810 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:11.476+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:11.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:11.476+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:11.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:11.476+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:11.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:11.477+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:12.100 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:11.926+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:12.100 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:11.926+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:12.101 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:11.926+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:12.101 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:11.927+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:12.466 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:12.103+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:12.466 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:12.103+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:12.467 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:12.103+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:12.467 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:12.104+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:12.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:12.470+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:12.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:12.470+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:12.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:12.470+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:12.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:12.471+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:13.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:12.902+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:13.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:12.902+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:13.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:12.902+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:13.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:12.903+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:13.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:13.087+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:13.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:13.087+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:13.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:13.087+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:13.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:13.088+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:13.851 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:13.490+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:13.851 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:13.490+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:13.851 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:13.490+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:13.851 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:13.490+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:14.117 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:13.853+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:14.117 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:13.854+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:14.117 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:13.854+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:14.117 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:13.855+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:14.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:14.120+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:14.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:14.120+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:14.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:14.120+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:14.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:14.121+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:14.810 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:14.534+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:14.810 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:14.534+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:14.810 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:14.534+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:14.810 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:14.535+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:15.069 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:14.813+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:15.069 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:14.813+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:15.069 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:14.813+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:15.069 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:14.814+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:15.470 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:01:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:01:15.131+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5679 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:01:15.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:15.072+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:15.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:15.072+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:15.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:15.072+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:15.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:15.073+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:15.830 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:15.833+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:15.830 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:15.833+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:15.830 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:15.833+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:15.830 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:15.833+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:15.831 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:15.583+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:15.831 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:15.583+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:15.831 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:15.583+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:15.831 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:15.584+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:16.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:16.116+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:16.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:16.116+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:16.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:16.116+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:16.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:16.117+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:16.970 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:16.785+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:16.970 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:16.785+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:16.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:16.785+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:16.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:16.786+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:16.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:16.588+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:16.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:16.588+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:16.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:16.588+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:16.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:16.589+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:17.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:17.120+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:17.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:17.120+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:17.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:17.120+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:17.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:17.121+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:17.815 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:17.632+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:17.816 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:17.632+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:17.816 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:17.632+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:17.816 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:17.633+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:18.136 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:17.819+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:18.136 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:17.819+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:18.136 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:17.819+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:18.136 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:17.820+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:18.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:18.139+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:18.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:18.139+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:18.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:18.139+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:18.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:18.140+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:18.970 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:18.828+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:18.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:18.828+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:18.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:18.828+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:18.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:18.829+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:18.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:18.599+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:18.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:18.599+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:18.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:18.599+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:18.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:18.600+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:19.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:19.125+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:19.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:19.125+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:19.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:19.125+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:19.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:19.125+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:19.873 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:19.567+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:19.874 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:19.567+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:19.874 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:19.567+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:19.874 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:19.568+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:20.129 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:19.877+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:20.129 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:19.877+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:20.129 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:19.877+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:20.129 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:19.877+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:20.470 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:01:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:01:20.131+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5775 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:01:20.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:20.170+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:20.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:20.170+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:20.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:20.170+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:20.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:20.171+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:20.865 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:20.611+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:20.865 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:20.611+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:20.865 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:20.611+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:20.865 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:20.612+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:21.156 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:20.867+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:21.156 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:20.867+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:21.156 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:20.867+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:21.156 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:20.868+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:21.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:21.158+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:21.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:21.158+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:21.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:21.158+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:21.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:21.159+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:21.895 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:21.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:21.895 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:21.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:21.895 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:21.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:21.896 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:21.627+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:22.184 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:21.897+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:22.184 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:21.897+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:22.184 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:21.897+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:22.184 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:21.898+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:22.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:22.186+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:22.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:22.186+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:22.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:22.186+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:22.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:22.187+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:22.935 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:22.647+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:22.935 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:22.647+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:22.935 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:22.647+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:22.935 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:22.648+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:23.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:22.937+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:23.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:22.937+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:23.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:22.937+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:23.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:22.938+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:23.538 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:23.228+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:23.538 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:23.228+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:23.538 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:23.228+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:23.538 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:23.228+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:23.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:23.635+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:23.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:23.635+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:23.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:23.635+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:23.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:23.635+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:24.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:24.201+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:24.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:24.201+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:24.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:24.201+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:24.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:24.202+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:24.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:23.976+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:24.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:23.976+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:24.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:23.976+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:24.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:23.976+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:24.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:24.593+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:24.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:24.593+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:24.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:24.593+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:24.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:24.593+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:25.252 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:01:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:01:25.132+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5870 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:01:25.253 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:25.173+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:25.253 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:25.173+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:25.253 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:25.173+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:25.253 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:25.174+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:25.253 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:24.975+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:25.253 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:24.975+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:25.253 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:24.975+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:25.253 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:24.975+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:25.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:25.574+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:25.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:25.574+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:25.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:25.574+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:25.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:25.575+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:26.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:25.972+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:26.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:25.972+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:26.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:25.972+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:26.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:25.973+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:26.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:26.221+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:26.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:26.221+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:26.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:26.221+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:26.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:26.221+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:26.960 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:26.561+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:26.960 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:26.561+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:26.960 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:26.561+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:26.960 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:26.563+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:27.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:26.962+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:27.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:26.962+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:27.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:26.962+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:27.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:26.963+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:27.541 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:27.224+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:27.541 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:27.224+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:27.541 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:27.224+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:27.542 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:27.225+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:27.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:27.544+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:27.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:27.544+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:27.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:27.544+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:27.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:27.544+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:28.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:27.985+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:28.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:27.985+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:28.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:27.985+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:28.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:27.986+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:28.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:28.230+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:28.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:28.230+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:28.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:28.230+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:28.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:28.231+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:28.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:28.540+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:28.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:28.540+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:28.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:28.540+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:28.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:28.540+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:29.243 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:29.210+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:29.243 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:29.210+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:29.243 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:29.210+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:29.243 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:29.210+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:29.243 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:28.985+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:29.243 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:28.985+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:29.243 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:28.985+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:29.244 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:28.986+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:29.939 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:29.557+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:29.939 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:29.557+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:29.940 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:29.557+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:29.940 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:29.558+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:30.220 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:01:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:01:30.133+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5963 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:01:30.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:30.185+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:30.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:30.185+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:30.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:30.185+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:30.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:30.186+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:30.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:29.941+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:30.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:29.941+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:30.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:29.941+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:30.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:29.943+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:30.968 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:30.522+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:30.968 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:30.522+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:30.969 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:30.522+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:30.969 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:30.523+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:31.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:30.970+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:31.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:30.970+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:31.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:30.971+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:31.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:30.971+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:31.654 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:31.230+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:31.654 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:31.230+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:31.654 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:31.230+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:31.654 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:31.231+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:31.654 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:31.475+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:31.654 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:31.475+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:31.655 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:31.475+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:31.655 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:31.475+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:32.203 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:31.944+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:32.203 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:31.944+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:32.203 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:31.944+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:32.203 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:31.944+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:32.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:32.205+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:32.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:32.205+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:32.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:32.205+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:32.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:32.206+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:32.922 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:32.500+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:32.922 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:32.500+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:32.922 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:32.500+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:32.922 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:32.501+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:33.208 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:32.924+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:33.209 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:32.924+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:33.209 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:32.924+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:33.209 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:32.925+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:33.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:33.211+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:33.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:33.211+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:33.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:33.211+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:33.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:33.211+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:33.875 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:33.522+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:33.875 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:33.522+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:33.875 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:33.522+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:33.875 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:33.523+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:34.181 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:33.877+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:34.182 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:33.877+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:34.182 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:33.877+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:34.182 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:33.878+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:34.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:34.184+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:34.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:34.184+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:34.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:34.184+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:34.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:34.184+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:34.917 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:34.513+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:34.918 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:34.513+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:34.918 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:34.513+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:34.918 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:34.514+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:35.201 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:01:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:01:35.133+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6059 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:01:35.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:34.920+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:35.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:34.920+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:35.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:34.920+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:35.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:34.921+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:35.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:35.203+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:35.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:35.203+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:35.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:35.203+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:35.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:35.204+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:35.960 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:35.553+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:35.960 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:35.553+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:35.960 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:35.553+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:35.960 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:35.553+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:36.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:35.963+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:36.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:35.963+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:36.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:35.963+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:36.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:35.963+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:36.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:36.174+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:36.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:36.174+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:36.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:36.174+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:36.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:36.175+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:36.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:36.587+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:36.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:36.587+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:36.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:36.587+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:36.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:36.588+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:37.216 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:36.918+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:37.216 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:36.918+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:37.216 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:36.918+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:37.217 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:36.919+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:37.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:37.219+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:37.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:37.219+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:37.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:37.219+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:37.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:37.219+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:37.943 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:37.552+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:37.943 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:37.552+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:37.943 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:37.552+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:37.943 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:37.553+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:38.211 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:37.945+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:38.211 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:37.945+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:38.211 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:37.945+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:38.211 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:37.945+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:38.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:38.214+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:38.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:38.214+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:38.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:38.214+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:38.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:38.214+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:38.902 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:38.582+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:38.902 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:38.582+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:38.903 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:38.582+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:38.903 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:38.582+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:39.205 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:38.904+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:39.206 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:38.904+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:39.206 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:38.904+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:39.206 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:38.905+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:39.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:39.208+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:39.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:39.208+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:39.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:39.208+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:39.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:39.209+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:39.861 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:39.539+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:39.862 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:39.539+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:39.862 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:39.539+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:39.862 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:39.540+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:40.132 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:39.865+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:40.132 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:39.865+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:40.132 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:39.865+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:40.133 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:39.865+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:40.470 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:01:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:01:40.135+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6154 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:01:40.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:40.231+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:40.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:40.231+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:40.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:40.231+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:40.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:40.232+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:40.895 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:40.491+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:40.895 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:40.491+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:40.895 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:40.491+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:40.895 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:40.491+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:41.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:40.898+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:41.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:40.898+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:41.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:40.898+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:41.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:40.899+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:41.495 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:41.498+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:41.495 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:41.498+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:41.495 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:41.248+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:41.495 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:41.248+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:41.495 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:41.248+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:41.495 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:41.249+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:41.942 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:41.498+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:41.942 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:41.499+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:42.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:41.945+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:42.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:41.945+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:42.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:41.945+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:42.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:41.946+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:42.519 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:42.252+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:42.519 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:42.252+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:42.520 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:42.252+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:42.520 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:42.253+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:42.912 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:42.522+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:42.912 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:42.522+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:42.913 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:42.522+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:42.913 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:42.523+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:43.203 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:42.915+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:43.203 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:42.915+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:43.203 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:42.915+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:43.203 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:42.916+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:43.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:43.206+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:43.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:43.206+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:43.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:43.206+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:43.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:43.207+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:43.864 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:43.570+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:43.865 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:43.570+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:43.865 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:43.570+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:43.865 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:43.570+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:44.161 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:43.868+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:44.161 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:43.868+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:44.161 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:43.868+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:44.161 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:43.868+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:44.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:44.164+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:44.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:44.164+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:44.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:44.164+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:44.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:44.165+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:44.843 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:44.522+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:44.843 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:44.522+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:44.843 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:44.522+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:44.843 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:44.522+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:45.134 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:44.846+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:45.134 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:44.846+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:45.134 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:44.846+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:45.134 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:44.847+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:45.470 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:01:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:01:45.136+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6249 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:01:45.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:45.140+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:45.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:45.140+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:45.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:45.140+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:45.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:45.142+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:45.887 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:45.500+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:45.887 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:45.500+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:45.888 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:45.500+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:45.888 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:45.501+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:46.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:45.891+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:46.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:45.891+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:46.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:45.891+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:46.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:45.891+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:46.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:46.137+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:46.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:46.137+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:46.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:46.137+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:46.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:46.139+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:46.931 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:46.549+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:46.931 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:46.550+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:46.931 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:46.550+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:46.931 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:46.550+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:47.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:46.934+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:47.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:46.934+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:47.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:46.934+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:47.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:46.935+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:47.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:47.179+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:47.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:47.179+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:47.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:47.179+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:47.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:47.179+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:47.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:47.531+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:47.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:47.531+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:47.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:47.531+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:47.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:47.532+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:48.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:47.903+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:48.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:47.903+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:48.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:47.903+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:48.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:47.904+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:48.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:48.137+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:48.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:48.137+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:48.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:48.137+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:48.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:48.137+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:48.935 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:48.488+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:48.935 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:48.488+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:48.935 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:48.488+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:48.935 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:48.489+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:49.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:48.938+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:49.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:48.938+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:49.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:48.938+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:49.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:48.939+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:49.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:49.118+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:49.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:49.118+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:49.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:49.118+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:49.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:49.118+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:49.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:49.487+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:49.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:49.488+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:49.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:49.488+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:49.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:49.488+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:50.256 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:01:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:01:50.137+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6343 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:01:50.257 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:49.987+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:50.257 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:49.987+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:50.257 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:49.987+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:50.257 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:49.987+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:50.257 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:50.078+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:50.257 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:50.078+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:50.258 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:50.078+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:50.258 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:50.078+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:50.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:50.461+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:50.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:50.461+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:50.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:50.461+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:50.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:50.462+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:51.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:50.970+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:51.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:50.970+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:51.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:50.970+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:51.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:50.970+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:51.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:51.102+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:51.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:51.102+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:51.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:51.102+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:51.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:51.103+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:51.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:51.419+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:51.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:51.419+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:51.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:51.419+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:51.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:51.420+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:52.113 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:51.972+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:52.113 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:51.972+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:52.113 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:51.972+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:52.113 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:51.973+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:52.406 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:52.116+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:52.407 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:52.116+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:52.407 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:52.116+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:52.407 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:52.117+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:52.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:52.409+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:52.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:52.409+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:52.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:52.409+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:52.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:52.410+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:53.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:52.944+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:53.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:52.944+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:53.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:52.944+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:53.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:52.945+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:53.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:53.125+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:53.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:53.125+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:53.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:53.125+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:53.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:53.126+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:53.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:53.381+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:53.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:53.381+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:53.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:53.381+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:53.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:53.382+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:54.405 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:54.125+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:54.405 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:54.125+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:54.405 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:54.125+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:54.405 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:54.125+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:54.405 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:53.977+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:54.405 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:53.977+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:54.405 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:53.977+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:54.405 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:53.977+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:54.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:54.408+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:54.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:54.408+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:54.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:54.408+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:54.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:54.409+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:55.135 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:54.984+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:55.135 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:54.984+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:55.135 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:54.984+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:55.135 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:54.985+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:55.387 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:01:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:01:55.138+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6439 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:01:55.387 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:55.157+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:55.387 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:55.157+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:55.388 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:55.157+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:55.388 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:55.158+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:55.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:55.390+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:55.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:55.390+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:55.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:55.390+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:55.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:55.391+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:56.383 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:56.019+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:56.383 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:56.019+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:56.383 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:56.019+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:56.384 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:56.020+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:56.384 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:56.187+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:56.384 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:56.187+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:56.384 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:56.187+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:56.384 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:56.187+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:56.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:56.386+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:56.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:56.386+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:56.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:56.386+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:56.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:56.387+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:57.344 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:57.050+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:57.345 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:57.050+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:57.345 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:57.050+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:57.345 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:57.050+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:57.345 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:57.235+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:57.345 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:57.235+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:57.345 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:57.235+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:57.345 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:57.236+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:57.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:57.347+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:57.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:57.347+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:57.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:57.347+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:57.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:57.348+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:58.304 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:58.003+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:58.304 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:58.003+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:58.305 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:58.003+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:58.305 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:58.003+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:58.305 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:58.232+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:58.305 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:58.232+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:58.305 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:58.232+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:58.305 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:58.233+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:58.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:58.307+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:58.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:58.307+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:58.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:58.307+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:58.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:58.308+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:59.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:58.970+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:59.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:58.970+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:59.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:58.970+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:01:59.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:01:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:01:58.971+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:01:59.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:59.186+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:59.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:59.186+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:59.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:59.186+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:01:59.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:01:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:01:59.187+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:01:59.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:59.302+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:59.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:59.302+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:59.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:59.302+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:01:59.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:01:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:01:59.303+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:00.339 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:02:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:02:00.139+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6535 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:02:00.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:00.017+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:00.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:00.017+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:00.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:00.017+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:00.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:00.018+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:00.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:00.192+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:00.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:00.192+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:00.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:00.192+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:00.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:00.194+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:00.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:00.343+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:00.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:00.343+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:00.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:00.343+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:00.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:00.343+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:01.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:01.054+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:01.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:01.054+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:01.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:01.054+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:01.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:01.055+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:01.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:01.295+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:01.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:01.295+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:01.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:01.296+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:01.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:01.296+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:01.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:01.166+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:01.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:01.166+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:01.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:01.166+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:01.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:01.167+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:02.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:02.034+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:02.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:02.034+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:02.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:02.034+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:02.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:02.034+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:02.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:02.182+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:02.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:02.182+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:02.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:02.182+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:02.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:02.183+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:02.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:02.278+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:02.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:02.278+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:02.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:02.278+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:02.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:02.279+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:03.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:03.225+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:03.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:03.225+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:03.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:03.225+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:03.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:03.225+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:03.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:03.062+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:03.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:03.062+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:03.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:03.062+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:03.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:03.063+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:03.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:03.248+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:03.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:03.248+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:03.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:03.248+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:03.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:03.249+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:04.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:04.016+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:04.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:04.016+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:04.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:04.016+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:04.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:04.017+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:04.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:04.236+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:04.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:04.236+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:04.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:04.236+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:04.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:04.237+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:04.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:04.230+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:04.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:04.230+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:04.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:04.230+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:04.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:04.231+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:05.470 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:02:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:02:05.140+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6629 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:02:05.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:05.229+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:05.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:05.229+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:05.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:05.229+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:05.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:05.229+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:05.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:05.040+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:05.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:05.040+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:05.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:05.040+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:05.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:05.041+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:05.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:05.239+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:05.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:05.239+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:05.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:05.239+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:05.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:05.240+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:06.266 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:06.194+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:06.266 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:06.194+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:06.266 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:06.194+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:06.266 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:06.195+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:06.267 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:05.997+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:06.267 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:05.997+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:06.267 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:05.997+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:06.267 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:05.998+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:06.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:06.269+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:06.269+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:06.269+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:06.270+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:07.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:07.171+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:07.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:07.171+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:07.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:07.171+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:07.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:07.172+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:07.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:06.952+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:07.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:06.952+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:07.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:06.952+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:07.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:06.953+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:07.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:07.290+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:07.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:07.290+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:07.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:07.290+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:07.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:07.291+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:08.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:07.937+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:08.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:07.937+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:08.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:07.937+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:08.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:07.938+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:08.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:08.149+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:08.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:08.149+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:08.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:08.149+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:08.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:08.150+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:08.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:08.282+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:08.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:08.282+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:08.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:08.282+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:08.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:08.283+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:09.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:09.153+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:09.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:09.153+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:09.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:09.153+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:09.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:09.153+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:09.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:08.956+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:09.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:08.956+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:09.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:08.956+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:09.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:08.957+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:09.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:09.260+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:09.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:09.260+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:09.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:09.260+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:09.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:09.260+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:10.261 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:02:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:02:10.141+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6724 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:02:10.262 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:09.985+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:10.262 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:09.985+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:10.262 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:09.985+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:10.262 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:09.986+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:10.262 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:10.112+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:10.262 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:10.112+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:10.262 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:10.112+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:10.262 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:10.113+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:10.263 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:10.213+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:10.263 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:10.213+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:10.263 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:10.213+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:10.263 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:10.213+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:11.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:11.006+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:11.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:11.006+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:11.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:11.006+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:11.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:11.007+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:11.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:11.139+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:11.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:11.139+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:11.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:11.139+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:11.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:11.139+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:11.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:11.174+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:11.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:11.174+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:11.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:11.174+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:11.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:11.175+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:12.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:12.108+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:12.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:12.108+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:12.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:12.108+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:12.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:12.109+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:12.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:11.976+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:12.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:11.976+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:12.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:11.976+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:12.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:11.977+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:12.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:12.136+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:12.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:12.136+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:12.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:12.136+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:12.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:12.137+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:13.251 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:13.089+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:13.251 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:13.089+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:13.251 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:13.089+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:13.251 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:13.090+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:13.251 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:12.976+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:13.251 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:12.976+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:13.251 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:12.976+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:13.252 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:12.977+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:13.252 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:13.171+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:13.252 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:13.171+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:13.252 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:13.171+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:13.252 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:13.171+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:14.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:13.975+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:14.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:13.975+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:14.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:13.975+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:14.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:13.976+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:14.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:14.131+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:14.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:14.131+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:14.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:14.131+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:14.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:14.132+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:14.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:14.206+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:14.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:14.206+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:14.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:14.206+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:14.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:14.206+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:15.220 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:02:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:02:15.142+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6822 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:02:15.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:14.960+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:15.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:14.960+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:15.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:14.960+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:15.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:14.960+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:15.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:15.091+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:15.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:15.091+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:15.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:15.091+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:15.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:15.092+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:15.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:15.200+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:15.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:15.200+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:15.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:15.200+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:15.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:15.201+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:16.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:16.051+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:16.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:16.051+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:16.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:16.051+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:16.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:16.052+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:15.918+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:15.918+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:15.918+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:15.919+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:16.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:16.224+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:16.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:16.224+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:16.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:16.224+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:16.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:16.224+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:17.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:17.008+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:17.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:17.008+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:17.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:17.008+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:17.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:17.008+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:17.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:16.916+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:17.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:16.916+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:17.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:16.916+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:17.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:16.917+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:17.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:17.254+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:17.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:17.254+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:17.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:17.254+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:17.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:17.254+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:18.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:18.005+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:18.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:18.005+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:18.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:18.005+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:18.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:18.006+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:18.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:17.891+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:18.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:17.891+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:18.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:17.891+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:18.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:17.892+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:18.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:18.235+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:18.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:18.235+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:18.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:18.235+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:18.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:18.236+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:19.214 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:18.888+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:19.215 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:18.888+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:19.215 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:18.888+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:19.215 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:18.888+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:19.215 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:18.992+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:19.215 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:18.992+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:19.215 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:18.992+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:19.215 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:18.993+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:19.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:19.218+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:19.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:19.218+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:19.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:19.218+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:19.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:19.218+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:20.185 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:02:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:02:20.143+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6917 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:02:20.185 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:19.986+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:20.185 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:19.986+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:20.185 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:19.986+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:20.185 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:19.987+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:20.185 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:19.924+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:20.186 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:19.924+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:20.186 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:19.924+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:20.186 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:19.925+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:20.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:20.188+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:51.590523+0000 front 2024-01-23T08:51:51.590423+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:20.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:20.188+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:51.590541+0000 front 2024-01-23T08:51:51.590554+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:20.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:20.188+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:51.590491+0000 front 2024-01-23T08:51:51.590476+0000 (oldest deadline 2024-01-23T08:52:13.890354+0000) 2024-01-23T09:02:20.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:20.189+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:21.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:20.951+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:21.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:20.951+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:21.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:20.951+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:21.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:20.952+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:21.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:21.027+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:21.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:21.027+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:21.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:21.027+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:21.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:21.027+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:21.658 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:21.225+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:22.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:21.972+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:22.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:21.973+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:22.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:21.973+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:22.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:21.974+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:22.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:22.037+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:22.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:22.037+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:22.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:22.037+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:22.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:22.038+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:22.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:22.267+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:23.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:23.243+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:23.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:22.994+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:23.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:22.994+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:23.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:22.994+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:23.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:22.995+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:23.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:23.028+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:23.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:23.028+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:23.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:23.028+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:23.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:23.028+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:24.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:24.207+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:24.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:24.004+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:24.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:24.004+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:24.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:24.004+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:24.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:24.005+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:24.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:24.070+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:24.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:24.070+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:24.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:24.070+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:24.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:24.071+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:25.220 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:02:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:02:25.144+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7013 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:02:25.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:25.159+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:25.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:24.971+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:25.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:24.971+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:25.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:24.971+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:25.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:24.972+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:25.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:25.077+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:25.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:25.077+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:25.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:25.077+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:25.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:25.078+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:25.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T09:02:25.916+0000 7f95c1ee3700 0 monclient(hunting): authenticate timed out after 300 2024-01-23T09:02:25.913 INFO:teuthology.orchestra.run.smithi059.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-01-23T09:02:26.220 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:26.144+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:26.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:25.942+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:26.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:25.942+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:26.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:25.942+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:26.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:25.943+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:26.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:26.127+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:26.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:26.127+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:26.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:26.127+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:26.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:26.128+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:26.666 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-01-23T09:02:26.940 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/mon.smithi059/config 2024-01-23T09:02:27.184 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:27.108+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:27.185 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:27.086+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:27.185 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:27.086+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:27.185 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:27.086+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:27.185 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:27.087+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:27.185 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:26.932+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:27.185 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:26.932+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:27.185 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:26.932+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:27.186 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:26.933+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:28.110 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:27.983+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:28.111 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:27.983+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:28.111 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:27.983+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:28.111 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:27.983+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:28.377 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:28.143+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:28.378 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:28.113+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:28.378 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:28.113+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:28.378 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:28.113+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:28.378 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:28.114+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:29.220 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:29.131+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:29.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:29.116+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:29.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:29.116+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:29.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:29.116+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:29.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:29.116+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:29.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:28.959+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:48.263224+0000 front 2024-01-23T08:51:48.263200+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:29.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:28.959+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:48.263144+0000 front 2024-01-23T08:51:48.263168+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:29.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:28.959+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:48.263134+0000 front 2024-01-23T08:51:48.263184+0000 (oldest deadline 2024-01-23T08:52:13.563147+0000) 2024-01-23T09:02:29.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:28.961+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:30.168 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:02:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:02:30.144+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7107 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:02:30.168 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:29.919+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:30.168 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:30.124+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:30.168 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:30.124+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:30.168 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:30.124+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:30.169 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:30.125+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:30.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:30.171+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:31.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:30.930+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:31.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:31.133+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:31.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:31.154+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:31.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:31.154+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:31.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:31.154+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:31.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:31.155+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:32.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:31.978+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:32.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:32.106+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:32.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:32.179+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:32.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:32.179+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:32.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:32.179+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:32.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:32.180+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:33.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:32.967+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:33.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:33.136+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:33.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:33.130+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:33.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:33.130+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:33.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:33.130+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:33.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:33.131+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:34.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:34.004+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:34.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:34.147+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:34.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:34.170+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 since back 2024-01-23T08:51:50.942841+0000 front 2024-01-23T08:51:50.942953+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:34.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:34.170+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 since back 2024-01-23T08:51:50.942939+0000 front 2024-01-23T08:51:50.942907+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:34.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:34.170+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 since back 2024-01-23T08:51:50.942894+0000 front 2024-01-23T08:51:50.942927+0000 (oldest deadline 2024-01-23T08:52:15.642964+0000) 2024-01-23T09:02:34.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:34.171+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:35.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:35.142+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:35.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:35.158+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:35.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:02:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:02:35.145+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7204 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:02:35.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:35.054+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:36.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:36.156+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:36.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:36.015+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:36.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:36.124+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:37.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:36.969+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:37.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:37.119+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:37.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:37.115+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:38.136 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:37.924+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:38.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:38.139+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:38.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:38.165+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:39.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:38.926+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:39.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:39.099+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:39.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:39.202+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:40.220 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:02:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:02:40.146+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7300 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:02:40.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:39.935+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:40.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:40.068+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:40.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:40.240+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:41.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:41.199+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:41.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:40.978+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:41.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:41.020+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:42.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:42.019+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:42.246 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:41.997+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:42.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:42.248+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:43.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:42.986+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:43.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:42.950+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:43.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:42.950+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:43.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:42.950+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:43.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:42.951+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:43.540 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:43.289+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:44.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:43.944+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:44.220 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:43.988+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:44.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:43.988+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:44.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:43.988+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:44.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:43.989+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:44.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:44.321+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:45.267 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:02:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:02:45.146+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7396 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:02:45.267 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:44.995+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:45.268 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:44.991+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:45.268 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:44.991+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:45.268 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:44.991+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:45.268 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:44.992+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:45.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:45.289+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:46.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:45.988+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:46.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:45.965+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:46.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:45.965+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:46.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:45.965+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:46.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:45.966+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:46.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:46.240+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:47.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:46.974+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:47.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:46.941+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:47.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:46.941+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:47.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:46.941+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:47.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:46.942+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:47.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:47.223+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:48.236 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:48.013+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:48.237 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:47.981+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:48.237 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:47.981+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:48.237 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:47.981+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:48.237 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:47.981+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:48.491 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:48.241+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:49.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:49.011+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:49.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:48.964+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:49.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:48.964+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:49.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:48.964+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:49.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:48.965+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:49.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:49.250+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:50.268 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:02:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:02:50.147+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7490 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:02:50.269 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:50.026+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:50.269 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:50.004+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:50.269 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:50.004+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:50.269 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:50.004+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:50.269 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:50.005+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:50.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:50.282+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:51.274 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:50.988+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:51.274 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:51.017+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:51.274 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:51.017+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:51.274 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:51.017+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:51.274 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:51.018+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:51.659 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:51.276+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:52.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:51.958+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:52.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:52.064+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:52.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:52.064+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:52.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:52.064+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:52.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:52.064+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:52.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:52.227+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:53.186 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:52.933+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:53.187 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:53.047+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:53.187 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:53.047+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:53.187 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:53.047+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:53.187 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:53.047+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:53.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:53.247+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:54.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:53.922+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:54.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:54.064+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:54.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:54.064+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:54.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:54.064+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:54.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:54.064+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:54.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:54.236+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:55.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:55.213+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:55.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:02:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:02:55.148+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7586 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:02:55.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:54.967+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:02:55.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:54.967+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:02:55.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:54.967+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:02:55.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:54.968+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:55.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:55.090+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:55.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:55.090+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:55.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:55.090+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:55.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:55.090+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:56.218 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:55.950+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:02:56.218 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:55.950+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:02:56.218 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:55.950+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:02:56.219 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:55.950+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:56.219 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:56.086+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:56.219 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:56.086+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:56.219 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:56.086+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:56.219 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:56.087+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:56.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:56.221+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:57.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:57.213+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:57.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:56.981+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:02:57.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:56.981+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:02:57.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:56.982+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:02:57.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:56.982+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:57.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:57.106+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:57.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:57.106+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:57.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:57.106+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:57.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:57.106+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:58.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:58.026+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:02:58.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:58.026+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:02:58.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:58.026+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:02:58.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:58.026+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:58.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:58.259+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:02:58.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:58.259+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:02:58.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:58.259+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:02:58.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:58.260+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:02:58.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:58.100+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:58.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:58.100+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:58.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:58.100+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:58.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:58.100+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:59.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:59.041+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:02:59.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:59.041+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:02:59.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:59.041+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:02:59.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:02:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:59.042+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:59.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:59.088+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:59.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:59.088+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:59.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:59.088+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:02:59.221 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:02:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:02:59.089+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:02:59.513 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:59.290+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:02:59.514 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:59.290+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:02:59.514 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:59.290+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:02:59.514 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:02:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:02:59.291+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:00.270 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:03:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:03:00.150+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7681 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:03:00.271 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:59.999+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:00.271 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:59.999+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:00.271 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:02:59.999+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:00.271 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:00.000+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:00.271 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:00.114+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:00.271 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:00.114+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:00.271 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:00.114+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:00.271 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:00.115+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:00.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:00.285+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:00.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:00.285+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:00.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:00.285+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:00.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:00.286+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:01.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:01.153+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:01.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:01.153+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:01.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:01.153+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:01.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:01.153+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:01.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:01.028+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:01.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:01.028+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:01.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:01.028+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:01.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:01.029+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:01.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:01.258+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:01.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:01.258+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:01.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:01.258+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:01.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:01.259+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:02.403 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:02.221+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:02.403 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:02.221+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:02.403 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:02.221+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:02.403 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:02.222+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:02.403 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:01.985+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:02.403 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:01.985+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:02.403 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:01.985+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:02.403 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:01.986+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:02.404 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:02.159+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:02.404 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:02.159+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:02.404 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:02.159+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:02.404 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:02.160+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:03.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:03.016+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:03.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:03.016+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:03.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:03.016+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:03.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:03.017+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:03.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:03.250+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:03.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:03.250+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:03.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:03.250+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:03.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:03.251+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:03.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:03.160+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:03.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:03.160+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:03.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:03.160+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:03.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:03.161+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:04.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:04.065+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:04.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:04.065+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:04.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:04.065+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:04.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:04.066+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:04.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:04.298+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:04.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:04.298+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:04.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:04.298+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:04.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:04.298+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:04.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:04.157+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:04.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:04.157+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:04.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:04.157+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:04.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:04.158+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:05.317 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:03:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:03:05.151+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7775 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:03:05.318 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:05.055+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:05.318 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:05.055+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:05.318 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:05.055+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:05.318 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:05.056+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:05.318 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:05.129+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:05.318 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:05.129+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:05.318 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:05.129+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:05.318 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:05.129+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:05.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:05.320+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:05.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:05.320+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:05.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:05.320+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:05.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:05.321+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:06.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:06.283+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:06.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:06.283+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:06.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:06.283+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:06.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:06.283+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:06.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:06.105+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:06.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:06.105+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:06.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:06.105+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:06.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:06.106+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:06.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:06.102+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:06.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:06.102+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:06.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:06.102+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:06.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:06.102+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:07.309 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:07.056+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:07.309 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:07.056+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:07.309 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:07.056+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:07.309 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:07.057+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:07.309 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:07.156+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:07.309 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:07.156+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:07.309 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:07.156+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:07.309 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:07.156+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:07.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:07.312+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:07.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:07.312+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:07.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:07.312+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:07.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:07.313+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:08.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:08.273+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:08.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:08.273+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:08.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:08.273+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:08.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:08.273+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:08.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:08.065+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:08.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:08.065+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:08.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:08.065+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:08.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:08.066+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:08.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:08.148+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:08.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:08.148+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:08.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:08.148+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:08.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:08.149+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:09.229 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:09.187+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:09.229 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:09.187+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:09.229 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:09.187+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:09.229 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:09.187+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:09.230 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:09.091+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:09.230 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:09.091+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:09.230 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:09.091+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:09.230 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:09.092+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:09.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:09.232+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:09.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:09.232+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:09.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:09.232+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:09.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:09.233+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:10.470 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:03:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:03:10.152+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7869 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:03:10.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:10.218+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:10.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:10.218+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:10.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:10.218+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:10.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:10.219+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:10.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:10.262+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:10.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:10.262+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:10.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:10.262+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:10.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:10.263+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:10.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:10.137+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:10.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:10.137+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:10.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:10.137+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:10.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:10.137+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:11.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:11.254+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:11.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:11.254+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:11.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:11.254+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:11.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:11.254+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:11.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:11.272+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:11.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:11.272+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:11.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:11.272+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:11.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:11.273+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:11.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:11.116+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:11.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:11.116+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:11.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:11.116+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:11.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:11.116+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:12.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:12.121+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:12.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:12.121+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:12.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:12.121+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:12.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:12.122+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:12.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:12.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:12.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:12.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:12.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:12.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:12.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:12.241+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:12.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:12.265+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:12.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:12.265+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:12.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:12.265+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:12.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:12.266+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:13.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:13.192+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:13.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:13.192+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:13.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:13.192+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:13.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:13.193+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:13.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:13.251+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:13.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:13.251+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:13.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:13.251+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:13.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:13.252+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:13.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:13.074+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:13.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:13.074+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:13.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:13.074+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:13.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:13.074+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:14.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:14.211+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:14.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:14.211+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:14.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:14.211+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:14.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:14.213+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:14.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:14.204+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:14.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:14.204+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:14.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:14.204+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:14.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:14.204+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:14.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:14.081+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:14.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:14.081+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:14.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:14.081+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:14.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:14.082+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:15.470 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:03:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:03:15.153+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7964 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:03:15.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:15.102+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:15.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:15.102+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:15.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:15.102+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:15.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:15.103+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:15.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:15.252+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:15.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:15.252+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:15.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:15.252+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:15.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:15.253+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:15.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:15.156+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:15.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:15.156+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:15.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:15.156+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:15.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:15.157+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:16.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:16.137+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:16.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:16.137+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:16.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:16.137+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:16.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:16.138+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:16.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:16.215+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:16.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:16.215+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:16.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:16.215+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:16.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:16.216+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:16.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:16.201+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:16.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:16.201+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:16.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:16.201+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:16.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:16.202+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:17.371 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:17.226+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:17.372 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:17.226+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:17.372 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:17.226+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:17.372 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:17.227+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:17.372 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:17.107+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:17.372 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:17.107+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:17.372 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:17.107+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:17.372 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:17.107+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:17.373 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:17.240+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:17.373 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:17.240+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:17.373 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:17.240+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:17.373 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:17.241+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:18.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:18.149+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:18.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:18.149+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:18.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:18.149+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:18.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:18.150+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:18.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:18.203+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:18.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:18.203+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:18.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:18.204+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:18.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:18.205+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:18.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:18.264+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:18.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:18.264+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:18.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:18.264+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:18.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:18.265+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:19.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:19.202+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:19.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:19.202+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:19.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:19.202+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:19.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:19.203+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:19.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:19.135+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:19.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:19.135+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:19.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:19.135+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:19.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:19.135+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:19.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:19.270+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:19.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:19.270+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:19.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:19.270+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:19.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:19.271+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:20.470 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:03:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:03:20.154+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8059 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:03:20.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:20.176+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:20.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:20.176+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:20.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:20.176+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:20.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:20.177+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:20.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:20.309+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:20.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:20.309+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:20.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:20.309+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:20.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:20.309+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:20.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:20.180+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:20.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:20.180+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:20.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:20.180+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:20.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:20.181+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:21.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:21.192+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:21.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:21.192+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:21.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:21.192+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:21.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:21.193+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:21.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:21.195+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:21.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:21.195+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:21.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:21.195+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:21.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:21.196+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:21.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:21.329+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:21.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:21.329+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:21.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:21.329+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:21.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:21.330+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:22.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:22.304+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:22.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:22.304+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:22.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:22.304+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:22.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:22.305+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:22.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:22.237+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:22.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:22.237+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:22.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:22.237+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:22.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:22.238+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:22.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:22.231+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:22.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:22.231+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:22.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:22.231+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:22.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:22.231+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:23.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:23.222+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:23.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:23.222+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:23.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:23.222+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:23.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:23.223+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:23.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:23.264+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:23.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:23.264+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:23.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:23.264+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:23.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:23.265+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:23.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:23.211+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:23.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:23.211+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:23.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:23.211+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:23.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:23.212+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:24.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:24.233+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:24.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:24.233+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:24.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:24.233+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:24.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:24.234+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:24.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:24.290+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:24.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:24.290+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:24.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:24.290+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:24.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:24.291+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:24.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:24.182+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:24.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:24.182+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:24.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:24.182+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:24.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:24.183+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:25.470 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:03:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:03:25.155+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8154 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:03:25.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:25.264+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:25.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:25.264+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:25.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:25.264+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:25.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:25.265+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:25.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:25.293+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:25.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:25.293+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:25.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:25.293+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:25.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:25.294+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:25.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:25.212+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:25.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:25.212+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:25.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:25.212+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:25.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:25.213+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:26.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:26.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:26.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:26.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:26.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:26.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:26.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:26.242+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:26.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:26.294+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:26.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:26.294+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:26.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:26.294+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:26.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:26.295+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:26.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:26.207+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:26.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:26.207+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:26.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:26.207+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:26.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:26.208+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:27.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:27.285+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:27.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:27.285+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:27.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:27.285+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:27.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:27.286+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:27.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:27.312+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:27.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:27.312+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:27.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:27.312+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:27.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:27.313+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:27.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:27.185+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:27.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:27.185+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:27.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:27.185+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:27.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:27.186+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:28.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:28.248+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:28.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:28.248+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:28.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:28.248+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:28.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:28.249+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:28.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:28.327+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:28.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:28.327+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:28.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:28.327+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:28.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:28.328+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:28.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:28.204+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:28.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:28.204+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:28.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:28.204+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:28.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:28.205+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:29.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:29.214+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:29.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:29.214+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:29.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:29.214+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:29.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:29.215+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:29.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:29.326+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:29.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:29.326+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:29.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:29.326+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:29.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:29.327+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:29.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:29.229+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:29.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:29.229+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:29.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:29.229+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:29.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:29.229+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:30.341 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:03:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:03:30.157+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8248 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:03:30.342 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:30.194+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:30.342 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:30.194+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:30.342 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:30.194+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:30.342 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:30.195+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:30.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:30.231+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:30.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:30.231+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:30.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:30.231+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:30.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:30.231+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:30.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:30.345+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:30.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:30.345+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:30.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:30.345+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:30.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:30.346+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:31.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:31.245+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:31.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:31.245+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:31.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:31.245+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:31.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:31.246+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:31.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:31.339+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:31.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:31.339+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:31.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:31.339+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:31.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:31.339+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:31.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:31.195+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:31.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:31.195+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:31.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:31.195+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:31.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:31.195+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:32.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:32.271+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:32.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:32.271+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:32.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:32.271+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:32.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:32.271+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:32.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:32.353+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:32.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:32.353+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:32.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:32.353+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:32.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:32.354+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:32.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:32.182+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:32.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:32.182+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:32.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:32.182+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:32.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:32.182+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:33.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:33.233+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:33.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:33.233+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:33.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:33.233+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:33.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:33.234+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:33.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:33.360+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:33.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:33.360+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:33.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:33.360+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:33.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:33.361+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:33.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:33.205+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:33.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:33.205+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:33.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:33.205+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:33.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:33.206+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:34.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:34.235+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:34.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:34.235+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:34.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:34.235+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:34.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:34.236+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:34.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:34.214+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:34.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:34.214+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:34.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:34.214+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:34.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:34.215+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:34.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:34.396+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:34.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:34.396+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:34.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:34.396+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:34.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:34.396+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:35.417 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:03:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:03:35.158+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8343 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:03:35.418 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:35.239+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:35.418 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:35.239+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:35.418 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:35.239+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:35.418 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:35.240+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:35.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:35.228+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:35.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:35.228+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:35.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:35.228+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:35.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:35.229+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:35.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:35.421+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:35.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:35.421+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:35.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:35.421+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:35.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:35.421+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:36.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:36.244+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:36.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:36.244+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:36.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:36.244+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:36.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:36.245+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:36.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:36.286+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:36.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:36.286+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:36.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:36.286+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:36.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:36.287+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:36.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:36.431+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:36.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:36.431+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:36.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:36.431+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:36.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:36.432+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:37.458 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:37.266+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:37.458 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:37.266+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:37.458 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:37.266+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:37.458 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:37.267+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:37.458 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:37.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:37.458 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:37.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:37.458 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:37.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:37.458 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:37.199+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:37.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:37.461+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:37.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:37.461+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:37.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:37.461+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:37.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:37.462+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:38.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:38.201+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:38.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:38.201+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:38.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:38.201+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:38.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:38.202+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:38.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:38.420+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:38.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:38.420+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:38.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:38.420+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:38.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:38.421+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:38.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:38.308+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:38.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:38.308+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:38.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:38.308+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:38.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:38.309+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:39.457 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:39.180+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:39.457 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:39.180+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:39.457 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:39.180+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:39.457 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:39.180+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:39.457 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:39.270+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:39.457 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:39.270+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:39.457 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:39.270+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:39.458 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:39.271+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:39.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:39.460+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:39.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:39.460+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:39.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:39.460+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:39.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:39.461+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:40.466 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:03:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:03:40.159+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8437 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:03:40.466 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:40.229+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:40.466 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:40.229+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:40.466 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:40.229+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:40.466 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:40.230+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:40.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:40.146+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:40.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:40.146+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:40.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:40.146+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:40.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:40.147+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:40.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:40.469+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:40.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:40.469+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:40.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:40.469+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:40.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:40.470+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:41.443 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:41.277+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:41.444 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:41.277+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:41.444 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:41.277+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:41.444 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:41.277+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:41.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:41.181+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:41.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:41.181+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:41.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:41.181+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:41.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:41.182+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:41.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:41.447+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:41.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:41.447+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:41.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:41.447+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:41.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:41.448+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:42.463 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:42.276+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:42.463 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:42.276+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:42.463 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:42.276+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:42.464 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:42.276+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:42.464 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:42.201+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:42.464 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:42.201+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:42.464 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:42.201+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:42.464 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:42.201+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:42.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:42.467+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:42.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:42.467+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:42.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:42.467+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:42.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:42.467+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:43.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:43.196+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:43.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:43.196+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:43.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:43.196+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:43.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:43.197+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:43.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:43.290+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:43.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:43.290+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:43.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:43.290+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:43.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:43.291+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:43.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:43.476+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:43.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:43.476+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:43.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:43.476+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:43.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:43.477+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:44.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:44.436+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:44.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:44.436+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:44.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:44.436+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:44.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:44.437+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:44.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:44.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:44.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:44.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:44.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:44.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:44.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:44.242+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:44.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:44.276+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:44.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:44.276+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:44.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:44.276+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:44.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:44.277+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:45.410 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:03:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:03:45.160+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8533 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:03:45.411 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:45.263+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:45.411 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:45.263+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:45.411 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:45.263+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:45.411 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:45.264+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:45.411 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:45.300+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:45.411 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:45.300+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:45.411 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:45.300+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:45.411 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:45.301+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:45.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:45.485+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:45.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:45.485+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:45.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:45.485+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:45.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:45.486+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:46.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:46.451+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:46.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:46.451+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:46.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:46.451+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:46.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:46.452+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:46.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:46.217+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:46.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:46.217+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:46.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:46.217+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:46.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:46.218+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:46.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:46.257+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:46.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:46.257+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:46.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:46.257+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:46.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:46.257+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:47.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:47.202+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:47.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:47.202+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:47.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:47.202+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:47.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:47.203+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:47.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:47.272+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:47.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:47.272+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:47.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:47.272+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:47.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:47.273+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:47.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:47.480+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:47.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:47.480+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:47.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:47.480+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:47.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:47.481+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:48.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:48.202+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:48.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:48.202+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:48.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:48.202+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:48.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:48.202+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:48.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:48.307+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:48.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:48.307+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:48.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:48.307+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:48.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:48.308+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:48.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:48.455+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:48.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:48.455+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:48.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:48.455+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:48.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:48.456+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:49.462 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:49.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:49.462 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:49.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:49.462 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:49.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:49.462 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:49.198+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:49.462 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:49.322+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:49.462 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:49.322+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:49.462 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:49.322+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:49.463 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:49.322+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:49.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:49.465+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:49.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:49.465+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:49.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:49.465+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:49.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:49.466+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:50.470 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:03:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:03:50.161+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8626 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:03:50.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:50.202+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:50.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:50.202+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:50.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:50.202+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:50.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:50.203+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:50.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:50.354+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:50.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:50.354+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:50.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:50.354+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:50.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:50.355+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:50.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:50.508+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:50.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:50.508+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:50.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:50.508+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:50.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:50.509+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:51.500 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:51.250+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:51.500 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:51.250+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:51.500 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:51.250+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:51.500 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:51.251+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:51.501 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:51.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:51.501 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:51.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:51.501 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:51.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:51.501 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:51.369+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:51.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:51.503+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:51.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:51.503+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:51.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:51.503+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:51.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:51.504+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:52.720 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:52.270+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:52.720 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:52.270+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:52.720 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:52.270+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:52.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:52.270+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:52.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:52.485+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:52.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:52.485+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:52.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:52.485+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:52.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:52.486+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:52.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:52.392+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:52.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:52.392+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:52.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:52.392+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:52.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:52.393+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:53.544 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:53.244+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:53.544 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:53.244+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:53.544 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:53.244+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:53.544 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:53.245+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:53.544 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:53.445+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:53.544 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:53.445+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:53.544 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:53.445+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:53.545 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:53.446+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:53.545 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:53.414+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:53.545 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:53.414+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:53.545 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:53.414+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:53.545 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:53.415+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:54.627 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:54.485+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:54.627 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:54.485+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:54.627 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:54.485+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:54.627 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:54.485+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:54.627 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:54.444+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:54.627 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:54.444+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:54.627 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:54.444+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:54.627 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:54.444+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:54.628 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:54.247+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:54.628 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:54.247+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:54.628 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:54.247+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:54.628 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:54.247+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:55.440 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:03:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:03:55.162+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8723 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:03:55.441 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:55.204+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:55.441 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:55.204+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:55.441 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:55.204+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:55.441 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:55.204+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:55.441 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:55.444+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:55.441 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:55.444+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:55.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:55.444+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:55.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:55.445+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:55.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:55.446+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:55.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:55.446+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:55.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:55.446+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:55.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:55.447+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:56.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:56.409+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:56.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:56.409+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:56.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:56.409+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:56.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:56.409+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:56.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:56.178+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:56.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:56.178+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:56.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:56.178+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:56.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:56.178+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:56.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:56.474+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:56.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:56.474+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:56.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:56.474+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:56.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:56.475+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:57.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:57.131+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:57.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:57.131+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:57.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:57.131+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:57.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:57.132+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:57.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:57.518+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:57.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:57.518+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:57.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:57.518+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:57.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:57.519+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:57.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:57.441+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:57.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:57.441+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:57.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:57.441+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:57.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:57.442+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:58.411 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:58.113+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:58.411 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:58.113+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:58.411 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:58.113+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:58.411 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:58.114+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:58.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:58.486+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:58.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:58.486+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:58.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:58.486+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:58.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:58.487+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:58.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:58.414+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:58.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:58.414+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:58.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:58.414+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:58.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:58.415+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:59.457 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:59.162+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:59.457 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:59.162+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:59.457 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:59.162+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:03:59.457 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:03:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:03:59.162+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:03:59.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:59.475+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:59.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:59.475+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:59.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:59.475+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:03:59.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:03:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:03:59.476+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:03:59.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:59.459+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:59.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:59.459+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:59.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:59.459+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:03:59.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:03:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:03:59.459+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:00.470 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:04:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:04:00.162+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8818 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:04:00.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:00.196+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:00.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:00.196+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:00.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:00.196+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:00.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:00.197+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:00.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:00.510+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:00.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:00.510+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:00.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:00.510+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:00.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:00.511+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:00.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:00.499+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:00.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:00.499+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:00.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:00.499+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:00.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:00.499+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:01.454 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:01.168+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:01.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:01.168+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:01.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:01.168+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:01.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:01.168+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:01.455 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:01.457+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:01.455 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:01.457+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:01.455 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:01.457+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:01.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:01.458+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:01.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:01.475+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:01.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:01.475+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:01.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:01.475+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:01.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:01.475+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:02.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:02.119+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:02.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:02.119+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:02.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:02.119+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:02.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:02.120+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:02.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:02.459+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:02.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:02.459+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:02.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:02.459+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:02.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:02.460+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:02.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:02.453+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:02.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:02.453+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:02.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:02.453+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:02.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:02.454+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:03.456 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:03.170+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:03.456 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:03.170+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:03.456 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:03.170+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:03.456 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:03.171+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:03.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:03.458+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:03.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:03.458+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:03.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:03.458+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:03.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:03.459+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:03.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:03.483+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:03.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:03.483+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:03.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:03.483+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:03.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:03.484+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:04.421 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:04.152+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:04.494 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:04.152+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:04.494 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:04.152+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:04.494 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:04.153+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:04.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:04.501+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:04.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:04.501+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:04.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:04.501+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:04.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:04.501+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:04.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:04.423+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:04.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:04.424+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:04.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:04.424+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:04.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:04.424+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:05.435 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:04:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:04:05.163+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8915 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:04:05.435 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:05.188+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:05.435 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:05.188+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:05.435 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:05.188+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:05.435 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:05.188+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:05.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:05.437+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:05.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:05.437+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:05.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:05.437+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:05.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:05.438+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:05.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:05.546+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:05.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:05.546+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:05.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:05.546+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:05.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:05.546+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:06.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:06.152+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:06.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:06.152+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:06.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:06.152+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:06.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:06.153+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:06.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:06.503+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:06.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:06.503+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:06.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:06.503+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:06.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:06.504+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:06.455+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:06.455+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:06.455+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:06.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:06.455+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:07.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:07.189+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:07.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:07.189+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:07.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:07.189+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:07.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:07.190+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:07.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:07.422+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:07.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:07.422+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:07.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:07.422+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:07.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:07.423+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:07.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:07.523+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:07.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:07.523+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:07.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:07.523+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:07.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:07.524+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:08.435 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:08.162+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:08.435 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:08.162+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:08.435 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:08.162+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:08.435 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:08.162+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:08.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:08.507+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:08.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:08.507+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:08.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:08.507+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:08.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:08.507+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:08.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:08.438+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:08.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:08.438+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:08.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:08.438+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:08.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:08.438+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:09.430 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:09.153+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:09.431 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:09.153+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:09.431 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:09.153+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:09.431 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:09.154+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:09.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:09.433+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:09.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:09.433+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:09.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:09.433+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:09.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:09.433+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:09.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:09.462+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:09.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:09.462+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:09.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:09.462+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:09.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:09.463+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:10.285 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:04:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:04:10.164+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9009 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:04:10.285 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:10.193+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:10.285 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:10.193+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:10.285 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:10.193+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:10.285 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:10.194+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:10.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:10.467+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:10.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:10.467+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:10.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:10.467+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:10.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:10.468+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:10.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:10.431+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:10.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:10.431+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:10.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:10.431+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:10.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:10.432+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:11.454 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:11.172+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:11.454 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:11.172+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:11.454 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:11.172+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:11.454 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:11.173+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:11.455 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:11.399+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:11.455 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:11.399+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:11.455 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:11.399+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:11.455 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:11.399+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:11.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:11.456+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:11.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:11.456+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:11.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:11.456+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:11.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:11.457+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:12.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:12.462+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:12.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:12.462+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:12.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:12.462+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:12.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:12.463+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:12.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:12.214+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:12.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:12.214+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:12.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:12.214+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:12.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:12.215+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:12.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:12.447+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:12.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:12.447+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:12.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:12.447+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:12.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:12.448+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:13.417 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:13.203+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:13.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:13.203+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:13.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:13.203+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:13.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:13.203+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:13.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:13.432+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:13.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:13.432+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:13.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:13.432+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:13.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:13.433+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:13.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:13.420+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:13.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:13.420+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:13.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:13.420+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:13.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:13.421+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:14.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:14.180+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:14.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:14.180+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:14.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:14.180+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:14.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:14.181+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:14.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:14.391+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:14.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:14.391+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:14.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:14.391+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:14.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:14.391+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:14.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:14.390+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:14.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:14.390+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:14.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:14.390+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:14.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:14.391+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:15.416 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:04:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:04:15.165+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9106 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:04:15.416 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:15.191+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:15.416 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:15.191+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:15.416 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:15.191+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:15.417 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:15.192+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:15.417 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:15.374+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:15.417 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:15.374+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:15.417 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:15.374+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:15.417 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:15.375+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:15.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:15.418+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:15.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:15.418+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:15.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:15.418+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:15.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:15.419+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:16.463 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:16.179+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:16.463 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:16.179+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:16.463 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:16.179+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:16.464 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:16.180+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:16.464 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:16.367+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:16.464 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:16.367+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:16.464 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:16.367+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:16.464 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:16.368+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:16.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:16.465+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:16.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:16.465+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:16.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:16.465+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:16.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:16.466+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:17.466 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:17.170+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:17.466 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:17.170+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:17.466 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:17.170+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:17.466 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:17.171+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:17.466 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:17.353+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:17.466 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:17.353+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:17.466 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:17.353+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:17.466 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:17.353+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:17.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:17.468+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:17.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:17.468+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:17.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:17.468+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:17.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:17.469+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:18.463 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:18.193+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:18.463 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:18.193+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:18.463 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:18.193+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:18.463 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:18.194+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:18.463 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:18.383+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:18.464 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:18.383+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:18.464 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:18.383+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:18.464 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:18.384+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:18.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:18.465+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:18.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:18.465+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:18.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:18.465+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:18.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:18.466+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:19.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:19.384+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:19.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:19.384+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:19.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:19.384+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:19.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:19.384+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:19.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:19.178+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:19.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:19.179+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:19.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:19.179+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:19.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:19.179+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:19.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:19.505+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:19.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:19.505+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:19.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:19.505+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:19.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:19.506+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:20.422 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:04:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:04:20.167+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9199 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:04:20.423 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:20.186+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:20.423 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:20.186+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:20.423 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:20.186+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:20.423 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:20.187+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:20.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:20.465+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:20.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:20.465+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:20.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:20.465+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:20.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:20.466+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:20.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:20.426+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:20.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:20.426+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:20.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:20.426+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:20.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:20.426+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:21.413 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:21.139+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:21.413 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:21.139+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:21.413 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:21.139+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:21.413 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:21.139+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:21.664 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:21.416+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:21.664 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:21.416+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:21.665 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:21.416+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:21.665 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:21.417+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:21.665 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:21.432+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:21.665 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:21.432+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:21.665 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:21.432+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:21.665 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:21.433+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:22.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:22.155+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:22.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:22.155+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:22.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:22.155+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:22.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:22.156+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:22.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:22.458+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:22.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:22.458+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:22.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:22.458+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:22.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:22.459+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:22.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:22.479+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:22.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:22.479+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:22.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:22.479+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:22.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:22.479+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:23.406 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:23.146+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:23.407 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:23.146+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:23.407 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:23.146+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:23.407 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:23.146+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:23.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:23.410+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:23.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:23.410+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:23.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:23.410+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:23.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:23.410+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:23.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:23.513+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:23.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:23.513+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:23.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:23.513+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:23.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:23.514+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:24.425 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:24.145+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:24.426 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:24.145+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:24.426 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:24.145+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:24.426 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:24.146+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:24.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:24.429+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:24.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:24.429+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:24.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:24.429+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:24.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:24.430+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:24.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:24.545+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:24.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:24.545+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:24.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:24.545+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:24.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:24.546+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:25.442 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:04:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:04:25.168+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9291 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:04:25.442 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:25.110+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:25.442 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:25.110+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:25.442 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:25.110+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:25.442 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:25.112+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:25.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:25.444+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:25.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:25.444+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:25.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:25.444+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:25.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:25.445+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:25.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:25.505+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:25.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:25.505+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:25.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:25.505+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:25.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:25.506+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:26.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:26.150+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:26.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:26.150+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:26.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:26.150+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:26.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:26.151+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:26.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:26.484+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:26.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:26.484+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:26.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:26.484+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:26.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:26.485+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:26.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:26.515+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:26.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:26.516+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:26.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:26.516+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:26.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:26.516+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:27.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:27.201+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:27.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:27.201+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:27.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:27.201+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:27.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:27.202+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:27.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:27.522+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:27.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:27.522+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:27.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:27.522+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:27.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:27.523+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:27.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:27.539+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:27.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:27.539+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:27.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:27.539+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:27.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:27.539+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:28.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:28.153+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:28.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:28.153+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:28.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:28.153+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:28.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:28.154+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:28.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:28.514+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:28.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:28.514+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:28.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:28.514+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:28.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:28.514+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:28.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:28.552+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:28.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:28.552+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:28.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:28.552+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:28.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:28.553+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:29.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:29.193+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:29.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:29.193+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:29.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:29.193+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:29.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:29.194+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:29.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:29.496+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:29.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:29.496+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:29.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:29.496+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:29.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:29.496+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:29.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:29.567+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:29.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:29.567+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:29.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:29.567+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:29.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:29.568+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:30.470 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:04:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:04:30.169+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9378 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:04:30.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:30.236+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:30.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:30.236+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:30.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:30.236+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:30.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:30.237+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:30.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:30.517+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:30.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:30.517+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:30.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:30.517+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:30.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:30.518+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:30.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:30.552+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:30.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:30.552+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:30.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:30.552+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:30.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:30.553+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:31.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:31.214+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:31.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:31.214+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:31.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:31.214+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:31.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:31.215+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:31.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:31.528+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:31.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:31.528+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:31.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:31.528+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:31.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:31.528+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:31.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:31.485+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:31.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:31.485+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:31.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:31.485+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:31.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:31.486+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:32.458 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:32.192+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:32.459 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:32.192+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:32.459 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:32.192+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:32.459 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:32.193+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:32.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:32.462+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:32.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:32.462+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:32.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:32.462+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:32.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:32.462+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:32.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:32.524+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:32.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:32.524+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:32.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:32.524+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:32.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:32.524+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:33.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:33.190+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:33.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:33.190+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:33.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:33.190+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:33.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:33.191+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:33.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:33.546+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:33.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:33.546+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:33.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:33.546+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:33.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:33.547+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:33.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:33.473+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:33.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:33.473+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:33.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:33.473+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:33.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:33.474+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:34.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:34.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:34.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:34.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:34.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:34.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:34.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:34.198+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:34.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:34.581+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:34.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:34.581+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:34.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:34.581+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:34.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:34.582+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:34.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:34.496+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:34.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:34.496+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:34.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:34.496+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:34.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:34.496+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:35.444 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:04:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:04:35.170+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9466 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:04:35.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:35.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:35.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:35.242+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:35.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:35.242+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:35.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:35.242+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:35.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:35.447+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:35.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:35.447+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:35.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:35.447+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:35.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:35.448+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:35.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:35.564+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:35.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:35.564+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:35.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:35.564+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:35.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:35.565+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:36.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:36.210+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:36.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:36.210+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:36.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:36.210+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:36.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:36.211+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:36.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:36.440+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:36.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:36.440+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:36.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:36.440+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:36.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:36.441+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:36.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:36.601+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:36.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:36.601+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:36.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:36.601+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:36.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:36.601+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:37.610 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:37.255+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:37.610 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:37.255+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:37.611 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:37.255+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:37.611 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:37.256+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:37.611 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:37.461+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:37.611 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:37.461+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:37.611 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:37.461+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:37.611 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:37.462+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:37.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:37.614+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:37.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:37.614+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:37.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:37.614+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:37.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:37.615+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:38.657 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:38.427+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:38.658 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:38.427+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:38.658 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:38.427+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:38.658 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:38.428+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:38.658 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:38.229+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:38.658 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:38.229+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:38.658 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:38.229+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:38.658 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:38.230+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:38.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:38.661+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:38.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:38.661+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:38.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:38.661+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:38.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:38.662+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:39.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:39.180+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:39.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:39.180+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:39.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:39.180+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:39.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:39.181+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:39.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:39.387+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:39.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:39.387+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:39.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:39.387+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:39.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:39.388+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:39.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:39.673+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:39.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:39.673+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:39.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:39.673+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:39.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:39.674+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:40.431 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:04:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:04:40.171+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9551 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:04:40.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:40.230+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:40.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:40.230+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:40.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:40.230+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:40.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:40.231+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:40.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:40.435+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:40.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:40.435+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:40.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:40.435+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:40.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:40.436+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:40.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:40.701+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:40.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:40.701+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:40.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:40.701+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:40.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:40.702+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:41.666 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:41.413+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:41.666 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:41.413+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:41.666 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:41.413+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:41.666 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:41.414+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:41.666 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:41.255+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:41.666 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:41.255+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:41.666 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:41.255+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:41.667 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:41.256+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:41.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:41.678+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:41.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:41.678+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:41.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:41.678+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:41.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:41.678+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:42.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:42.414+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:42.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:42.414+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:42.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:42.414+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:42.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:42.415+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:42.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:42.222+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:42.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:42.222+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:42.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:42.222+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:42.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:42.222+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:42.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:42.690+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:42.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:42.690+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:42.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:42.690+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:42.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:42.691+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:43.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:43.395+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:43.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:43.396+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:43.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:43.396+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:43.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:43.397+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:43.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:43.195+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:43.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:43.195+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:43.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:43.195+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:43.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:43.196+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:43.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:43.662+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:43.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:43.662+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:43.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:43.662+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:43.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:43.663+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:44.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:44.221+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:44.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:44.221+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:44.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:44.221+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:44.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:44.221+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:44.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:44.440+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:44.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:44.440+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:44.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:44.440+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:44.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:44.441+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:44.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:44.623+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:44.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:44.623+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:44.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:44.623+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:44.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:44.624+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:45.430 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:04:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:04:45.173+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9641 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:04:45.431 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:45.184+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:45.431 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:45.184+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:45.431 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:45.184+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:45.431 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:45.185+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:45.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:45.575+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:45.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:45.575+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:45.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:45.575+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:45.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:45.576+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:45.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:45.434+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:45.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:45.434+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:45.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:45.434+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:45.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:45.435+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:46.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:46.222+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:46.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:46.222+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:46.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:46.222+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:46.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:46.223+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:46.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:46.434+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:46.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:46.434+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:46.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:46.434+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:46.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:46.435+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:46.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:46.596+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:46.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:46.596+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:46.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:46.596+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:46.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:46.597+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:47.600 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:47.239+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:47.600 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:47.239+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:47.600 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:47.239+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:47.600 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:47.240+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:47.601 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:47.413+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:47.601 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:47.413+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:47.601 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:47.413+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:47.601 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:47.414+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:47.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:47.603+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:47.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:47.603+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:47.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:47.603+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:47.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:47.604+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:48.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:48.219+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:48.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:48.219+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:48.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:48.219+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:48.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:48.219+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:48.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:48.435+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:48.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:48.435+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:48.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:48.435+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:48.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:48.436+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:48.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:48.616+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:48.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:48.616+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:48.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:48.616+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:48.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:48.617+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:49.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:49.183+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:49.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:49.183+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:49.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:49.183+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:49.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:49.184+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:49.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:49.634+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:49.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:49.634+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:49.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:49.634+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:49.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:49.635+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:49.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:49.455+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:49.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:49.455+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:49.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:49.455+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:49.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:49.456+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:50.465 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:04:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:04:50.174+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9728 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:04:50.466 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:50.181+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:50.466 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:50.181+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:50.466 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:50.181+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:50.466 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:50.183+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:50.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:50.469+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:50.808 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:50.469+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:50.808 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:50.469+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:50.809 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:50.470+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:50.809 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:50.654+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:50.809 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:50.654+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:50.809 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:50.654+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:50.809 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:50.655+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:51.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:51.166+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:51.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:51.166+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:51.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:51.166+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:51.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:51.167+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:51.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:51.474+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:51.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:51.474+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:51.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:51.474+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:51.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:51.475+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:51.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:51.684+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:51.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:51.684+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:51.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:51.684+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:51.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:51.685+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:52.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:52.140+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:52.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:52.140+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:52.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:52.140+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:52.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:52.141+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:52.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:52.690+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:52.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:52.690+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:52.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:52.690+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:52.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:52.691+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:52.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:52.522+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:52.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:52.522+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:52.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:52.522+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:52.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:52.522+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:53.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:53.151+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:53.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:53.151+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:53.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:53.151+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:53.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:53.152+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:53.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:53.715+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:53.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:53.715+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:53.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:53.715+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:53.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:53.716+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:53.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:53.475+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:53.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:53.475+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:53.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:53.475+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:53.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:53.475+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:54.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:54.159+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:54.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:54.159+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:54.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:54.159+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:54.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:54.159+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:54.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:54.709+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:54.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:54.709+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:54.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:54.709+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:54.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:54.709+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:54.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:54.491+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:54.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:54.491+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:54.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:54.491+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:54.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:54.491+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:55.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:04:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:04:55.175+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9822 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:04:55.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:55.126+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:55.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:55.126+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:55.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:55.126+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:55.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:55.127+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:55.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:55.746+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:55.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:55.746+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:55.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:55.746+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:55.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:55.746+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:55.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:55.497+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:55.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:55.497+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:55.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:55.497+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:55.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:55.497+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:56.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:56.082+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:56.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:56.082+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:56.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:56.082+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:56.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:56.083+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:56.784 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:56.517+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:56.784 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:56.517+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:56.784 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:56.517+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:56.784 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:56.517+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:57.058 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:56.787+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:57.058 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:56.787+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:57.059 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:56.787+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:57.059 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:56.788+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:57.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:57.062+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:57.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:57.062+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:57.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:57.062+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:57.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:57.062+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:57.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:57.543+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:57.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:57.543+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:57.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:57.543+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:57.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:57.544+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:57.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:57.742+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:57.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:57.742+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:57.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:57.742+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:57.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:57.743+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:58.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:58.048+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:58.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:58.048+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:58.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:58.048+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:58.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:58.048+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:58.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:58.762+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:58.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:58.762+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:58.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:58.762+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:58.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:58.763+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:58.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:58.551+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:58.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:58.551+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:58.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:58.551+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:58.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:58.552+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:04:59.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:59.089+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:59.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:59.089+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:59.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:59.089+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:04:59.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:04:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:04:59.090+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:59.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:59.728+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:59.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:59.728+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:59.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:59.728+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:04:59.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:04:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:04:59.729+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:04:59.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:59.570+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:59.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:59.570+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:59.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:59.570+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:04:59.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:04:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:04:59.571+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:00.470 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:05:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:05:00.176+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9916 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:05:00.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:00.132+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:00.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:00.132+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:00.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:00.132+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:00.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:00.133+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:00.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:00.526+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:00.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:00.526+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:00.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:00.526+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:00.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:00.526+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:00.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:00.705+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:00.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:00.705+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:00.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:00.705+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:00.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:00.706+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:01.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:01.156+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:01.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:01.156+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:01.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:01.156+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:01.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:01.157+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:01.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:01.554+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:01.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:01.554+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:01.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:01.554+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:01.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:01.554+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:01.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:01.756+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:01.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:01.756+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:01.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:01.756+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:01.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:01.756+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:02.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:02.198+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:02.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:02.198+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:02.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:02.198+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:02.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:02.199+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:02.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:02.552+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:02.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:02.552+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:02.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:02.552+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:02.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:02.553+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:02.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:02.752+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:02.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:02.752+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:02.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:02.752+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:02.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:02.752+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:03.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:03.181+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:03.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:03.181+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:03.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:03.181+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:03.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:03.182+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:03.797 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:03.800+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:03.797 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:03.800+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:03.797 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:03.800+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:03.797 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:03.550+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:03.797 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:03.550+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:03.797 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:03.550+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:03.797 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:03.550+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:04.191 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:03.800+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:04.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:04.194+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:04.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:04.194+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:04.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:04.194+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:04.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:04.195+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:04.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:04.757+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:04.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:04.757+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:04.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:04.757+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:04.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:04.757+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:04.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:04.577+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:04.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:04.577+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:04.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:04.577+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:04.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:04.578+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:05.470 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:05:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:05:05.177+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10012 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:05:05.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:05.222+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:05.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:05.222+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:05.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:05.222+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:05.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:05.223+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:05.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:05.757+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:05.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:05.757+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:05.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:05.757+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:05.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:05.758+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:05.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:05.595+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:05.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:05.595+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:05.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:05.595+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:05.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:05.596+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:06.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:06.176+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:06.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:06.176+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:06.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:06.176+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:06.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:06.176+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:06.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:06.713+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:06.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:06.713+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:06.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:06.713+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:06.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:06.714+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:06.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:06.585+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:06.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:06.585+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:06.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:06.585+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:06.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:06.585+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:07.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:07.135+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:07.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:07.135+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:07.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:07.135+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:07.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:07.136+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:07.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:07.597+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:07.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:07.597+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:07.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:07.597+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:07.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:07.598+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:07.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:07.679+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:07.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:07.679+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:07.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:07.679+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:07.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:07.680+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:08.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:08.124+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:08.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:08.124+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:08.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:08.124+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:08.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:08.125+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:08.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:08.640+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:08.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:08.640+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:08.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:08.640+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:08.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:08.640+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:08.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:08.657+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:08.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:08.657+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:08.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:08.657+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:08.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:08.658+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:09.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:09.078+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:09.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:09.078+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:09.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:09.079+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:09.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:09.080+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:09.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:09.640+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:09.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:09.640+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:09.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:09.640+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:09.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:09.641+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:09.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:09.608+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:09.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:09.608+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:09.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:09.608+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:09.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:09.609+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:10.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:05:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:05:10.178+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10105 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:05:10.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:10.113+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:10.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:10.113+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:10.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:10.113+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:10.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:10.114+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:10.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:10.588+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:10.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:10.588+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:10.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:10.588+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:10.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:10.589+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:10.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:10.621+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:10.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:10.621+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:10.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:10.621+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:10.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:10.621+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:11.427 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:11.078+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:11.428 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:11.078+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:11.428 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:11.078+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:11.428 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:11.078+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:11.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:11.577+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:11.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:11.577+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:11.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:11.577+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:11.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:11.578+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:11.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:11.657+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:11.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:11.657+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:11.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:11.657+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:11.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:11.658+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:12.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:12.039+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:12.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:12.039+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:12.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:12.039+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:12.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:12.040+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:12.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:12.601+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:12.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:12.601+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:12.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:12.601+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:12.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:12.601+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:12.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:12.664+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:12.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:12.664+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:12.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:12.664+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:12.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:12.664+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:13.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:13.018+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:13.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:13.018+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:13.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:13.018+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:13.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:13.019+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:13.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:13.573+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:13.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:13.573+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:13.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:13.573+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:13.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:13.574+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:13.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:13.690+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:13.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:13.690+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:13.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:13.690+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:13.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:13.690+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:14.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:13.985+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:14.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:13.985+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:14.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:13.985+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:14.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:13.985+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:14.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:14.594+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:14.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:14.594+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:14.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:14.594+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:14.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:14.594+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:14.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:14.647+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:14.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:14.647+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:14.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:14.647+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:14.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:14.648+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:15.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:05:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:05:15.180+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10203 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:05:15.300 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:15.007+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:15.300 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:15.007+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:15.300 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:15.007+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:15.300 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:15.008+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:15.683 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:15.551+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:15.683 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:15.551+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:15.683 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:15.551+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:15.683 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:15.552+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:15.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:15.686+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:15.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:15.686+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:15.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:15.686+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:15.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:15.686+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:16.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:16.005+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:16.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:16.005+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:16.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:16.005+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:16.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:16.006+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:16.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:16.589+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:16.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:16.589+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:16.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:16.589+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:16.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:16.590+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:16.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:16.641+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:16.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:16.641+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:16.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:16.641+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:16.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:16.641+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:17.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:17.013+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:17.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:17.013+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:17.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:17.013+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:17.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:17.014+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:17.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:17.618+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:17.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:17.618+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:17.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:17.618+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:17.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:17.618+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:17.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:17.554+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:17.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:17.554+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:17.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:17.554+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:17.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:17.555+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:18.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:18.058+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:18.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:18.058+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:18.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:18.058+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:18.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:18.058+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:18.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:18.667+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:18.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:18.667+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:18.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:18.667+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:18.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:18.668+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:18.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:18.508+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:18.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:18.508+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:18.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:18.508+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:18.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:18.509+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:19.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:19.074+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:19.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:19.074+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:19.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:19.074+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:19.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:19.075+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:19.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:19.532+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:19.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:19.532+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:19.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:19.532+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:19.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:19.533+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:19.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:19.649+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:19.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:19.649+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:19.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:19.649+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:19.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:19.650+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:20.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:05:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:05:20.180+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10298 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:05:20.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:20.061+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:20.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:20.061+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:20.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:20.061+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:20.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:20.062+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:20.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:20.618+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:20.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:20.618+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:20.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:20.618+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:20.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:20.618+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:20.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:20.512+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:20.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:20.512+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:20.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:20.512+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:20.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:20.512+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:21.466 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:21.052+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:21.466 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:21.052+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:21.466 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:21.052+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:21.466 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:21.053+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:21.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:21.629+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:21.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:21.629+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:21.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:21.629+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:21.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:21.629+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:21.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:21.468+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:21.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:21.468+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:21.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:21.468+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:21.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:21.469+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:22.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:22.046+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:22.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:22.046+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:22.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:22.046+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:22.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:22.046+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:22.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:22.494+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:22.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:22.494+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:22.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:22.494+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:22.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:22.495+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:22.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:22.679+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:22.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:22.679+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:22.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:22.679+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:22.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:22.680+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:23.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:23.021+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:23.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:23.021+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:23.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:23.021+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:23.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:23.022+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:23.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:23.495+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:23.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:23.495+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:23.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:23.495+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:23.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:23.496+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:23.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:23.707+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:23.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:23.707+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:23.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:23.707+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:23.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:23.708+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:24.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:24.055+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:24.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:24.055+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:24.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:24.055+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:24.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:24.056+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:24.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:24.541+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:24.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:24.541+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:24.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:24.541+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:24.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:24.542+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:24.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:24.684+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:24.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:24.684+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:24.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:24.684+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:24.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:24.684+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:25.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:05:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:05:25.181+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10396 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:05:25.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:25.071+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:25.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:25.071+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:25.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:25.071+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:25.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:25.072+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:25.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:25.563+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:25.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:25.563+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:25.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:25.563+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:25.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:25.564+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:25.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:25.698+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:25.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:25.698+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:25.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:25.698+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:25.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:25.699+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:26.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:26.121+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:26.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:26.121+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:26.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:26.121+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:26.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:26.122+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:26.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:26.582+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:26.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:26.582+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:26.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:26.582+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:26.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:26.583+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:26.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:26.672+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:26.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:26.672+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:26.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:26.672+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:26.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:26.672+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:27.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:27.117+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:27.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:27.117+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:27.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:27.117+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:27.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:27.117+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:27.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:27.624+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:27.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:27.624+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:27.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:27.624+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:27.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:27.625+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:27.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:27.576+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:27.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:27.576+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:27.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:27.576+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:27.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:27.576+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:28.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:28.095+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:28.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:28.095+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:28.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:28.095+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:28.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:28.096+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:28.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:28.662+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:28.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:28.662+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:28.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:28.662+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:28.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:28.663+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:28.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:28.562+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:28.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:28.562+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:28.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:28.562+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:28.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:28.562+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:29.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:29.087+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:29.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:29.087+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:29.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:29.087+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:29.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:29.088+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:29.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:29.576+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:29.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:29.576+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:29.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:29.576+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:29.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:29.577+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:29.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:29.671+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:29.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:29.671+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:29.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:29.671+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:29.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:29.671+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:30.302 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:05:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:05:30.182+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10491 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:05:30.303 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:30.040+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:30.303 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:30.040+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:30.303 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:30.040+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:30.303 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:30.042+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:30.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:30.657+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:30.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:30.657+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:30.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:30.657+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:30.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:30.657+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:30.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:30.569+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:30.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:30.569+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:30.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:30.569+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:30.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:30.570+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:31.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:31.056+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:31.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:31.056+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:31.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:31.056+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:31.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:31.057+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:31.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:31.636+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:31.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:31.636+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:31.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:31.636+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:31.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:31.636+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:31.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:31.560+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:31.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:31.560+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:31.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:31.560+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:31.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:31.561+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:32.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:32.026+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:32.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:32.026+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:32.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:32.026+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:32.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:32.026+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:32.931 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:32.511+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:32.931 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:32.511+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:32.931 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:32.511+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:32.931 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:32.512+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:32.932 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:32.660+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:32.932 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:32.660+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:32.932 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:32.660+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:32.932 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:32.661+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:33.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:33.074+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:33.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:33.074+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:33.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:33.074+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:33.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:33.075+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:33.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:33.536+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:33.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:33.536+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:33.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:33.536+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:33.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:33.537+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:33.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:33.643+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:33.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:33.643+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:33.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:33.643+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:33.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:33.644+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:34.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:34.036+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:34.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:34.036+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:34.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:34.036+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:34.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:34.037+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:34.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:34.489+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:34.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:34.489+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:34.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:34.489+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:34.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:34.490+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:34.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:34.601+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:34.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:34.601+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:34.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:34.601+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:34.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:34.602+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:35.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:05:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:05:35.183+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10590 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:05:35.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:35.085+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:35.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:35.085+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:35.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:35.085+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:35.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:35.085+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:35.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:35.576+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:35.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:35.576+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:35.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:35.576+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:35.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:35.576+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:35.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:35.527+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:35.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:35.527+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:35.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:35.527+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:35.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:35.528+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:36.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:36.110+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:36.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:36.110+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:36.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:36.110+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:36.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:36.111+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:36.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:36.574+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:36.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:36.574+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:36.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:36.574+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:36.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:36.574+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:36.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:36.490+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:36.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:36.490+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:36.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:36.490+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:36.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:36.490+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:37.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:37.104+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:37.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:37.104+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:37.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:37.104+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:37.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:37.105+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:37.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:37.522+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:37.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:37.522+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:37.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:37.522+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:37.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:37.523+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:37.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:37.542+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:37.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:37.542+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:37.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:37.542+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:37.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:37.542+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:38.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:38.105+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:38.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:38.105+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:38.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:38.105+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:38.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:38.106+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:38.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:38.532+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:38.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:38.532+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:38.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:38.532+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:38.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:38.533+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:38.485+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:38.485+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:38.485+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:38.485+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:39.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:39.101+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:39.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:39.101+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:39.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:39.101+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:39.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:39.102+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:39.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:39.545+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:39.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:39.545+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:39.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:39.545+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:39.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:39.546+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:39.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:39.510+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:39.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:39.510+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:39.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:39.510+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:39.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:39.511+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:40.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:05:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:05:40.186+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10686 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:05:40.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:40.113+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:40.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:40.113+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:40.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:40.113+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:40.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:40.114+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:40.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:40.589+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:40.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:40.589+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:40.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:40.589+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:40.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:40.590+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:40.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:40.499+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:40.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:40.499+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:40.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:40.499+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:40.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:40.500+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:41.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:41.164+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:41.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:41.164+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:41.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:41.164+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:41.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:41.165+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:41.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:41.543+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:41.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:41.543+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:41.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:41.543+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:41.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:41.544+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:41.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:41.508+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:41.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:41.508+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:41.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:41.508+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:41.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:41.509+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:42.469 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:42.116+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:42.469 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:42.116+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:42.469 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:42.116+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:42.469 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:42.116+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:42.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:42.472+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:42.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:42.472+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:42.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:42.472+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:42.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:42.472+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:42.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:42.506+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:42.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:42.506+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:42.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:42.506+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:42.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:42.507+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:43.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:43.150+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:43.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:43.150+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:43.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:43.150+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:43.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:43.150+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:43.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:43.480+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:43.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:43.480+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:43.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:43.480+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:43.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:43.480+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:43.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:43.507+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:43.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:43.507+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:43.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:43.507+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:43.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:43.507+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:44.433 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:44.148+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:44.433 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:44.148+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:44.433 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:44.148+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:44.433 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:44.148+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:44.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:44.554+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:44.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:44.554+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:44.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:44.554+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:44.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:44.556+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:44.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:44.435+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:44.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:44.435+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:44.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:44.435+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:44.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:44.436+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:45.436 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:05:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:05:45.187+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10783 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:05:45.436 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:45.123+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:45.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:45.123+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:45.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:45.123+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:45.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:45.124+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:45.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:45.439+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:45.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:45.439+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:45.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:45.439+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:45.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:45.440+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:45.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:45.558+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:45.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:45.558+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:45.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:45.558+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:45.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:45.559+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:46.466 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:46.076+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:46.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:46.076+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:46.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:46.076+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:46.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:46.077+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:46.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:46.469+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:46.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:46.469+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:46.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:46.469+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:46.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:46.470+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:46.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:46.522+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:46.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:46.522+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:46.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:46.522+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:46.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:46.523+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:47.460 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:47.120+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:47.460 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:47.120+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:47.460 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:47.120+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:47.460 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:47.120+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:47.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:47.462+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:47.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:47.462+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:47.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:47.462+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:47.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:47.463+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:47.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:47.541+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:47.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:47.541+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:47.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:47.541+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:47.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:47.542+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:48.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:48.167+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:48.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:48.167+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:48.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:48.167+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:48.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:48.168+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:48.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:48.575+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:48.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:48.575+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:48.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:48.575+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:48.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:48.575+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:48.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:48.480+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:48.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:48.480+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:48.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:48.480+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:48.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:48.481+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:49.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:49.165+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:49.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:49.165+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:49.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:49.165+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:49.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:49.166+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:49.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:49.506+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:49.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:49.506+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:49.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:49.506+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:49.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:49.507+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:49.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:49.586+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:49.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:49.586+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:49.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:49.586+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:49.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:49.587+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:50.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:05:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:05:50.188+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10878 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:05:50.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:50.194+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:50.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:50.194+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:50.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:50.194+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:50.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:50.196+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:50.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:50.477+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:50.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:50.477+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:50.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:50.477+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:50.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:50.478+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:50.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:50.573+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:50.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:50.573+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:50.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:50.573+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:50.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:50.574+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:51.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:51.444+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:51.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:51.444+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:51.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:51.444+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:51.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:51.444+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:51.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:51.216+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:51.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:51.216+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:51.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:51.216+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:51.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:51.217+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:51.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:51.556+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:51.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:51.556+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:51.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:51.556+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:51.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:51.556+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:52.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:52.176+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:52.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:52.176+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:52.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:52.176+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:52.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:52.177+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:52.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:52.554+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:52.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:52.554+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:52.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:52.554+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:52.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:52.555+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:52.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:52.449+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:52.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:52.449+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:52.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:52.449+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:52.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:52.449+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:53.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:53.167+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:53.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:53.167+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:53.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:53.167+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:53.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:53.168+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:53.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:53.598+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:53.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:53.598+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:53.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:53.598+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:53.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:53.598+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:53.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:53.489+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:53.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:53.489+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:53.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:53.489+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:53.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:53.489+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:54.435 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:54.157+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:54.435 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:54.157+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:54.435 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:54.157+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:54.435 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:54.158+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:54.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:54.592+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:54.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:54.592+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:54.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:54.592+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:54.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:54.592+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:54.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:54.449+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:54.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:54.449+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:54.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:54.449+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:54.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:54.450+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:55.417 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:05:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:05:55.189+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10977 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:05:55.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:55.131+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:55.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:55.131+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:55.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:55.131+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:55.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:55.132+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:55.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:55.616+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:55.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:55.616+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:55.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:55.616+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:55.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:55.616+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:55.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:55.420+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:55.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:55.420+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:55.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:55.420+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:55.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:55.421+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:56.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:56.180+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:56.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:56.180+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:56.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:56.180+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:56.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:56.181+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:56.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:56.435+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:56.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:56.435+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:56.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:56.435+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:56.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:56.436+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:56.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:56.663+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:56.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:56.663+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:56.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:56.663+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:56.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:56.664+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:57.409 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:57.149+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:57.409 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:57.149+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:57.409 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:57.149+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:57.409 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:57.150+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:57.675 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:57.412+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:57.675 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:57.412+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:57.676 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:57.412+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:57.676 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:57.413+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:57.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:57.678+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:57.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:57.678+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:57.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:57.678+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:57.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:57.679+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:58.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:58.176+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:58.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:58.176+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:58.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:58.176+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:58.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:58.177+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:58.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:58.417+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:58.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:58.417+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:58.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:58.417+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:58.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:58.418+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:05:58.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:58.689+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:58.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:58.689+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:58.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:58.689+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:58.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:58.690+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:59.462 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:59.154+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:59.462 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:59.154+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:59.462 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:59.154+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:05:59.462 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:05:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:05:59.155+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:59.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:59.682+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:59.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:59.682+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:59.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:59.682+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:05:59.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:05:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:05:59.683+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:05:59.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:59.464+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:59.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:59.464+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:59.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:59.464+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:05:59.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:05:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:05:59.465+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:00.452 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:06:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:06:00.190+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11073 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:06:00.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:00.155+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:00.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:00.155+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:00.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:00.155+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:00.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:00.156+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:00.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:00.653+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:00.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:00.653+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:00.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:00.653+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:00.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:00.654+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:00.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:00.455+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:00.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:00.455+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:00.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:00.455+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:00.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:00.456+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:01.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:01.189+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:01.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:01.189+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:01.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:01.189+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:01.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:01.190+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:01.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:01.481+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:01.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:01.481+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:01.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:01.481+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:01.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:01.482+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:01.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:01.664+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:01.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:01.664+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:01.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:01.664+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:01.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:01.665+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:02.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:02.190+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:02.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:02.190+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:02.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:02.190+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:02.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:02.190+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:02.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:02.532+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:02.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:02.532+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:02.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:02.532+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:02.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:02.532+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:02.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:02.623+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:02.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:02.623+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:02.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:02.623+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:02.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:02.624+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:03.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:03.160+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:03.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:03.160+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:03.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:03.160+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:03.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:03.161+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:03.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:03.489+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:03.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:03.489+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:03.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:03.489+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:03.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:03.490+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:03.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:03.591+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:03.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:03.591+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:03.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:03.591+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:03.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:03.592+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:04.461 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:04.122+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:04.462 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:04.122+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:04.462 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:04.122+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:04.462 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:04.122+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:04.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:04.464+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:04.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:04.464+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:04.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:04.464+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:04.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:04.465+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:04.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:04.551+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:04.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:04.551+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:04.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:04.551+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:04.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:04.552+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:05.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:06:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:06:05.191+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11170 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:06:05.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:05.075+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:05.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:05.075+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:05.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:05.075+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:05.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:05.076+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:05.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:05.507+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:05.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:05.507+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:05.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:05.507+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:05.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:05.508+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:05.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:05.510+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:05.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:05.510+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:05.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:05.510+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:05.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:05.511+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:06.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:06.065+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:06.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:06.065+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:06.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:06.065+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:06.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:06.066+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:06.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:06.505+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:06.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:06.505+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:06.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:06.505+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:06.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:06.506+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:06.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:06.496+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:06.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:06.496+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:06.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:06.496+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:06.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:06.496+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:07.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:07.018+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:07.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:07.018+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:07.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:07.018+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:07.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:07.018+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:07.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:07.551+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:07.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:07.551+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:07.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:07.551+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:07.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:07.552+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:07.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:07.501+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:07.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:07.501+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:07.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:07.501+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:07.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:07.502+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:08.469 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:08.038+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:08.469 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:08.038+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:08.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:08.038+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:08.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:08.039+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:08.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:08.472+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:08.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:08.472+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:08.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:08.472+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:08.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:08.473+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:08.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:08.590+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:08.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:08.590+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:08.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:08.590+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:08.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:08.592+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:09.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:09.052+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:09.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:09.052+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:09.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:09.052+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:09.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:09.053+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:09.796 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:09.639+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:09.796 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:09.639+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:09.796 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:09.639+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:09.796 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:09.640+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:09.796 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:09.500+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:09.796 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:09.500+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:09.797 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:09.500+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:09.797 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:09.501+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:10.313 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:06:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:06:10.193+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11265 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:06:10.314 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:10.052+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:10.314 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:10.052+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:10.314 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:10.052+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:10.314 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:10.053+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:10.658 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:10.484+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:10.659 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:10.484+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:10.659 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:10.485+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:10.659 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:10.485+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:10.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:10.661+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:10.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:10.661+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:10.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:10.661+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:10.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:10.662+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:11.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:11.048+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:11.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:11.048+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:11.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:11.048+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:11.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:11.049+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:11.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:11.492+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:11.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:11.492+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:11.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:11.492+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:11.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:11.492+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:11.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:11.687+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:11.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:11.687+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:11.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:11.687+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:11.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:11.687+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:12.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:12.098+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:12.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:12.098+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:12.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:12.098+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:12.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:12.098+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:12.867 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:12.697+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:12.867 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:12.697+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:12.868 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:12.697+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:12.868 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:12.698+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:12.868 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:12.493+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:12.868 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:12.493+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:12.868 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:12.493+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:12.868 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:12.494+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:13.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:13.127+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:13.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:13.127+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:13.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:13.127+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:13.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:13.128+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:13.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:13.448+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:13.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:13.448+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:13.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:13.448+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:13.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:13.448+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:14.080 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:13.738+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:14.080 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:13.738+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:14.080 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:13.738+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:14.080 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:13.738+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:14.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:14.082+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:14.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:14.082+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:14.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:14.082+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:14.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:14.083+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:14.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:14.702+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:14.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:14.702+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:14.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:14.702+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:14.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:14.703+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:14.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:14.492+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:14.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:14.493+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:14.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:14.493+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:14.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:14.493+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:15.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:06:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:06:15.194+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11364 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:06:15.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:15.131+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:15.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:15.131+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:15.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:15.131+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:15.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:15.132+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:15.939 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:15.534+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:15.939 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:15.534+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:15.940 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:15.534+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:15.940 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:15.535+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:15.940 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:15.727+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:15.940 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:15.727+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:15.940 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:15.727+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:15.940 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:15.728+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:16.178+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:16.178+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:16.178+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:16.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:16.179+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:16.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:16.777+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:16.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:16.777+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:16.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:16.777+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:16.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:16.778+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:16.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:16.563+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:16.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:16.563+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:16.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:16.563+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:16.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:16.564+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:17.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:17.202+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:17.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:17.202+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:17.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:17.202+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:17.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:17.203+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:17.773 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:17.598+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:17.773 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:17.598+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:17.773 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:17.598+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:17.773 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:17.598+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:18.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:17.775+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:18.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:17.775+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:18.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:17.775+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:18.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:17.776+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:18.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:18.236+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:18.560 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:18.236+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:18.560 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:18.236+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:18.560 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:18.237+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:18.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:18.746+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:18.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:18.746+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:18.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:18.746+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:18.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:18.747+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:18.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:18.562+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:18.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:18.562+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:18.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:18.562+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:18.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:18.563+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:19.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:19.244+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:19.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:19.244+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:19.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:19.244+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:19.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:19.244+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:19.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:19.724+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:19.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:19.724+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:19.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:19.724+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:19.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:19.724+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:19.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:19.557+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:19.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:19.557+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:19.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:19.557+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:19.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:19.558+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:20.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:06:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:06:20.195+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11460 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:06:20.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:20.263+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:20.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:20.263+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:20.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:20.263+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:20.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:20.264+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:20.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:20.698+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:20.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:20.698+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:20.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:20.698+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:20.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:20.698+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:20.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:20.531+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:20.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:20.531+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:20.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:20.531+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:20.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:20.532+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:21.565 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:21.238+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:21.565 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:21.238+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:21.565 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:21.238+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:21.565 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:21.239+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:21.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:21.666+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:21.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:21.666+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:21.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:21.666+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:21.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:21.666+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:21.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:21.567+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:21.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:21.567+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:21.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:21.567+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:21.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:21.568+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:22.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:22.274+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:22.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:22.274+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:22.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:22.274+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:22.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:22.275+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:22.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:22.623+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:23.112 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:22.623+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:23.112 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:22.623+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:23.113 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:22.624+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:23.113 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:22.574+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:23.113 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:22.574+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:23.113 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:22.574+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:23.113 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:22.574+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:23.553 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:23.238+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:23.553 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:23.238+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:23.554 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:23.238+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:23.554 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:23.239+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:23.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:23.643+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:23.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:23.643+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:23.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:23.643+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:23.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:23.643+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:23.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:23.580+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:23.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:23.580+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:23.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:23.580+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:23.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:23.581+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:24.571 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:24.259+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:24.571 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:24.259+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:24.571 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:24.259+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:24.571 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:24.260+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:24.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:24.680+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:24.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:24.680+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:24.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:24.680+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:24.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:24.681+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:24.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:24.574+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:24.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:24.574+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:24.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:24.574+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:24.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:24.574+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:25.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:06:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:06:25.196+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11557 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:06:25.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:25.295+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:25.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:25.295+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:25.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:25.295+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:25.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:25.295+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:25.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:25.719+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:25.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:25.719+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:25.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:25.719+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:25.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:25.720+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:25.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:25.545+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:25.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:25.545+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:25.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:25.545+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:25.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:25.546+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:26.675 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:26.291+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:26.675 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:26.291+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:26.675 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:26.291+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:26.675 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:26.291+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:26.675 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:26.517+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:26.675 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:26.517+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:26.675 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:26.517+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:26.675 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:26.518+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:26.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:26.677+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:26.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:26.677+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:26.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:26.677+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:26.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:26.678+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:27.656 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:27.286+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:27.656 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:27.286+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:27.656 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:27.286+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:27.657 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:27.287+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:27.657 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:27.483+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:27.657 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:27.483+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:27.657 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:27.483+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:27.657 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:27.484+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:27.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:27.659+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:27.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:27.659+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:27.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:27.659+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:27.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:27.660+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:28.659 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:28.280+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:28.659 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:28.280+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:28.659 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:28.280+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:28.659 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:28.281+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:28.659 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:28.473+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:28.659 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:28.473+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:28.659 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:28.473+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:28.660 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:28.474+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:28.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:28.661+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:28.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:28.661+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:28.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:28.661+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:28.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:28.662+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:29.611 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:29.290+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:29.611 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:29.290+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:29.612 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:29.290+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:29.612 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:29.291+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:29.612 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:29.443+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:29.612 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:29.443+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:29.612 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:29.443+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:29.612 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:29.444+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:29.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:29.614+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:29.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:29.614+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:29.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:29.614+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:29.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:29.615+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:30.442 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:06:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:06:30.197+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11650 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:06:30.443 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:30.280+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:30.443 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:30.280+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:30.443 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:30.280+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:30.443 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:30.280+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:30.443 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:30.445+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:30.444 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:30.445+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:30.444 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:30.445+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:30.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:30.446+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:30.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:30.586+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:30.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:30.586+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:30.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:30.586+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:30.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:30.587+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:31.602 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:31.245+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:31.602 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:31.245+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:31.603 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:31.245+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:31.603 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:31.246+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:31.603 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:31.460+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:31.603 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:31.460+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:31.603 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:31.460+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:31.603 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:31.460+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:31.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:31.605+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:31.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:31.605+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:31.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:31.605+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:31.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:31.606+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:32.503 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:32.231+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:32.503 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:32.231+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:32.503 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:32.231+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:32.503 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:32.231+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:32.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:32.626+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:32.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:32.626+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:32.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:32.626+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:32.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:32.626+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:32.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:32.505+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:32.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:32.505+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:32.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:32.505+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:32.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:32.506+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:33.519 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:33.226+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:33.519 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:33.226+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:33.519 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:33.226+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:33.519 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:33.226+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:33.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:33.621+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:33.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:33.621+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:33.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:33.621+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:33.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:33.622+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:33.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:33.521+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:33.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:33.521+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:33.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:33.521+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:33.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:33.522+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:34.533 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:34.240+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:34.534 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:34.240+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:34.534 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:34.240+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:34.534 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:34.241+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:34.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:34.612+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:34.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:34.612+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:34.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:34.612+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:34.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:34.612+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:34.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:34.536+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:34.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:34.536+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:34.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:34.536+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:34.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:34.536+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:35.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:06:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:06:35.199+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11747 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:06:35.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:35.245+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:35.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:35.245+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:35.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:35.245+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:35.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:35.245+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:35.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:35.627+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:35.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:35.627+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:35.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:35.627+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:35.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:35.628+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:35.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:35.523+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:35.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:35.523+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:35.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:35.523+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:35.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:35.524+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:36.537 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:36.246+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:36.537 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:36.246+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:36.537 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:36.246+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:36.537 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:36.246+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:36.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:36.582+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:36.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:36.582+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:36.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:36.582+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:36.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:36.583+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:36.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:36.539+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:36.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:36.539+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:36.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:36.539+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:36.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:36.540+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:37.547 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:37.258+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:37.547 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:37.258+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:37.547 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:37.258+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:37.547 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:37.259+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:37.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:37.571+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:37.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:37.571+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:37.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:37.571+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:37.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:37.571+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:37.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:37.549+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:37.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:37.549+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:37.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:37.549+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:37.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:37.550+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:38.554 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:38.236+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:38.554 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:38.236+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:38.554 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:38.236+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:38.555 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:38.237+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:38.599+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:38.599+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:38.599+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:38.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:38.599+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:38.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:38.556+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:38.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:38.556+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:38.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:38.556+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:38.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:38.557+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:39.571 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:39.236+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:39.571 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:39.236+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:39.571 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:39.236+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:39.571 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:39.236+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:39.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:39.639+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:39.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:39.639+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:39.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:39.639+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:39.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:39.641+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:39.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:39.572+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:39.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:39.572+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:39.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:39.572+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:39.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:39.573+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:40.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:06:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:06:40.199+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11843 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:06:40.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:40.230+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:40.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:40.230+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:40.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:40.230+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:40.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:40.231+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:40.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:40.660+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:40.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:40.660+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:40.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:40.660+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:40.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:40.661+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:40.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:40.556+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:40.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:40.556+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:40.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:40.556+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:40.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:40.557+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:41.578 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:41.230+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:41.578 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:41.230+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:41.578 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:41.230+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:41.578 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:41.231+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:41.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:41.636+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:41.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:41.636+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:41.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:41.636+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:41.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:41.637+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:41.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:41.579+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:41.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:41.579+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:41.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:41.579+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:41.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:41.580+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:42.604 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:42.273+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:42.604 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:42.273+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:42.604 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:42.273+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:42.605 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:42.273+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:42.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:42.606+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:42.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:42.606+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:42.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:42.606+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:42.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:42.607+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:42.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:42.648+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:42.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:42.648+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:42.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:42.648+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:42.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:42.649+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:43.555 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:43.226+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:43.555 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:43.226+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:43.555 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:43.226+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:43.555 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:43.227+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:43.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:43.617+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:43.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:43.617+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:43.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:43.617+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:43.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:43.618+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:43.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:43.580+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:43.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:43.580+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:43.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:43.580+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:43.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:43.581+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:44.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:44.207+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:44.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:44.207+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:44.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:44.207+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:44.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:44.208+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:44.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:44.572+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:44.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:44.572+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:44.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:44.572+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:44.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:44.573+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:44.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:44.650+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:44.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:44.650+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:44.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:44.650+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:44.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:44.651+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:45.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:06:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:06:45.200+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11939 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:06:45.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:45.202+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:45.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:45.202+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:45.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:45.202+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:45.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:45.203+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:45.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:45.602+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:45.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:45.602+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:45.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:45.602+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:45.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:45.603+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:45.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:45.691+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:45.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:45.691+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:45.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:45.691+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:45.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:45.692+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:46.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:46.218+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:46.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:46.218+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:46.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:46.218+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:46.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:46.219+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:46.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:46.712+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:46.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:46.712+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:46.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:46.712+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:46.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:46.713+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:46.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:46.596+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:46.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:46.596+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:46.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:46.596+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:46.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:46.597+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:47.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:47.169+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:47.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:47.169+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:47.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:47.169+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:47.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:47.170+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:47.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:47.683+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:47.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:47.683+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:47.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:47.683+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:47.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:47.684+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:47.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:47.588+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:47.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:47.588+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:47.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:47.588+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:47.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:47.588+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:48.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:48.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:48.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:48.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:48.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:48.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:48.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:48.198+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:48.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:48.670+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:48.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:48.670+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:48.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:48.670+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:48.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:48.671+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:48.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:48.589+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:48.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:48.589+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:48.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:48.589+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:48.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:48.589+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:49.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:49.230+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:49.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:49.230+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:49.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:49.230+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:49.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:49.231+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:49.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:49.693+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:49.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:49.693+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:49.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:49.693+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:49.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:49.694+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:49.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:49.578+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:49.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:49.578+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:49.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:49.578+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:49.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:49.578+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:50.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:06:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:06:50.201+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12034 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:06:50.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:50.226+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:50.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:50.226+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:50.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:50.226+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:50.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:50.227+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:50.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:50.737+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:50.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:50.737+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:50.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:50.737+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:50.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:50.738+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:50.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:50.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:50.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:50.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:50.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:50.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:50.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:50.626+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:51.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:51.239+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:51.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:51.239+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:51.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:51.239+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:51.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:51.240+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:51.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:51.698+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:51.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:51.698+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:51.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:51.698+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:51.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:51.698+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:51.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:51.632+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:51.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:51.632+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:51.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:51.632+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:51.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:51.633+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:52.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:52.201+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:52.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:52.201+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:52.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:52.201+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:52.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:52.202+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:52.869 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:52.667+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:52.870 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:52.667+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:52.870 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:52.667+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:52.870 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:52.667+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:52.870 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:52.616+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:52.870 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:52.616+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:52.870 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:52.616+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:52.870 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:52.616+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:53.508 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:53.249+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:53.508 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:53.249+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:53.508 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:53.249+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:53.508 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:53.250+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:53.810 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:53.649+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:53.810 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:53.649+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:53.810 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:53.649+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:53.810 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:53.650+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:53.810 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:53.595+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:53.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:53.595+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:53.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:53.595+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:53.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:53.596+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:54.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:54.297+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:54.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:54.297+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:54.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:54.297+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:54.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:54.298+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:54.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:54.639+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:54.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:54.639+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:54.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:54.639+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:54.703 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:54.640+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:54.703 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:54.622+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:54.703 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:54.622+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:54.703 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:54.622+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:54.703 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:54.622+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:55.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:06:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:06:55.203+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12131 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:06:55.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:55.299+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:55.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:55.299+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:55.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:55.299+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:55.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:55.300+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:55.875 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:55.612+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:55.876 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:55.612+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:55.876 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:55.612+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:55.876 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:55.613+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:55.876 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:55.609+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:55.876 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:55.609+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:55.876 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:55.609+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:55.876 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:55.610+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:56.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:56.293+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:56.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:56.293+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:56.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:56.293+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:56.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:56.294+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:56.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:56.575+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:56.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:56.575+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:56.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:56.575+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:56.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:56.575+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:56.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:56.621+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:56.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:56.621+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:56.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:56.621+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:56.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:56.621+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:57.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:57.337+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:57.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:57.337+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:57.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:57.337+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:57.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:57.338+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:57.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:57.598+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:57.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:57.598+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:57.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:57.598+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:57.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:57.599+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:57.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:57.593+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:57.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:57.593+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:57.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:57.593+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:57.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:57.594+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:58.721 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:58.372+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:58.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:58.372+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:58.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:58.372+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:58.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:58.373+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:58.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:58.552+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:58.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:58.552+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:58.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:58.552+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:58.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:58.553+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:58.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:58.606+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:58.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:58.606+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:58.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:58.606+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:58.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:58.606+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:59.592 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:59.343+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:59.592 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:59.343+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:59.592 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:59.343+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:06:59.592 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:06:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:06:59.343+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:06:59.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:59.595+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:59.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:59.595+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:59.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:59.595+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:06:59.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:06:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:06:59.595+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:06:59.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:59.649+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:59.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:59.649+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:59.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:59.649+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:06:59.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:06:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:06:59.650+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:00.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:07:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:07:00.205+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12226 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:07:00.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:00.306+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:00.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:00.306+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:00.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:00.306+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:00.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:00.307+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:00.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:00.553+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:00.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:00.553+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:00.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:00.553+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:00.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:00.553+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:00.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:00.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:00.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:00.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:00.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:00.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:00.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:00.627+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:01.575 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:01.577+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:01.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:01.327+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:01.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:01.328+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:01.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:01.328+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:01.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:01.328+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:01.576 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:01.522+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:01.576 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:01.522+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:01.576 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:01.522+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:01.576 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:01.523+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:01.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:01.577+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:01.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:01.577+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:01.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:01.578+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:02.557 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:02.279+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:02.558 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:02.279+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:02.558 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:02.279+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:02.558 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:02.280+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:02.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:02.583+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:02.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:02.583+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:02.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:02.583+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:02.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:02.584+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:02.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:02.560+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:02.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:02.560+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:02.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:02.560+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:02.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:02.561+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:03.518 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:03.256+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:03.518 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:03.256+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:03.518 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:03.256+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:03.518 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:03.257+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:03.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:03.520+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:03.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:03.520+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:03.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:03.520+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:03.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:03.521+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:03.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:03.543+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:03.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:03.543+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:03.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:03.543+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:03.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:03.544+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:04.563 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:04.287+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:04.563 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:04.287+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:04.563 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:04.287+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:04.563 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:04.288+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:04.563 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:04.536+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:04.563 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:04.536+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:04.563 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:04.536+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:04.564 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:04.537+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:04.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:04.565+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:04.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:04.565+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:04.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:04.565+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:04.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:04.566+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:05.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:07:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:07:05.206+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12321 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:07:05.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:05.254+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:05.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:05.254+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:05.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:05.254+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:05.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:05.255+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:05.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:05.607+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:05.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:05.607+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:05.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:05.607+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:05.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:05.608+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:05.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:05.490+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:05.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:05.490+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:05.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:05.490+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:05.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:05.491+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:06.628 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:06.520+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:06.628 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:06.520+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:06.628 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:06.520+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:06.628 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:06.521+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:06.629 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:06.290+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:06.629 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:06.290+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:06.629 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:06.290+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:06.629 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:06.291+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:06.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:06.630+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:06.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:06.630+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:06.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:06.630+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:06.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:06.631+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:07.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:07.338+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:07.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:07.338+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:07.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:07.338+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:07.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:07.338+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:07.632 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:07.543+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:07.632 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:07.543+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:07.632 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:07.543+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:07.632 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:07.544+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:07.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:07.633+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:07.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:07.633+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:07.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:07.633+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:07.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:07.634+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:08.651 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:08.305+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:08.651 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:08.305+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:08.651 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:08.305+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:08.651 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:08.306+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:08.651 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:08.508+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:08.651 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:08.508+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:08.651 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:08.508+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:08.651 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:08.509+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:08.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:08.653+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:08.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:08.653+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:08.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:08.653+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:08.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:08.654+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:09.556 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:09.283+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:09.556 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:09.283+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:09.556 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:09.283+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:09.556 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:09.283+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:09.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:09.661+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:09.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:09.661+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:09.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:09.661+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:09.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:09.662+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:09.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:09.558+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:09.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:09.558+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:09.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:09.558+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:09.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:09.559+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:10.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:07:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:07:10.208+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12416 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:07:10.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:10.291+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:10.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:10.291+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:10.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:10.291+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:10.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:10.292+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:10.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:10.680+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:10.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:10.680+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:10.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:10.680+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:10.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:10.681+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:10.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:10.587+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:10.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:10.587+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:10.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:10.587+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:10.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:10.588+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:11.596 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:11.264+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:11.596 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:11.264+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:11.596 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:11.264+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:11.596 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:11.264+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:11.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:11.598+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:11.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:11.598+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:11.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:11.598+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:11.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:11.599+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:11.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:11.651+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:11.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:11.651+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:11.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:11.651+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:11.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:11.652+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:12.551 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:12.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:12.551 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:12.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:12.551 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:12.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:12.552 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:12.242+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:12.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:12.701+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:12.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:12.701+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:12.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:12.701+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:12.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:12.701+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:12.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:12.554+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:12.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:12.554+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:12.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:12.554+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:12.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:12.554+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:13.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:13.210+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:13.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:13.210+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:13.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:13.210+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:13.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:13.210+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:13.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:13.731+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:13.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:13.732+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:13.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:13.732+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:13.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:13.732+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:13.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:13.529+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:13.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:13.529+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:13.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:13.529+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:13.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:13.529+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:14.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:14.209+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:14.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:14.209+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:14.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:14.209+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:14.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:14.210+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:14.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:14.764+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:14.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:14.764+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:14.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:14.764+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:14.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:14.765+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:14.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:14.538+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:14.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:14.538+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:14.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:14.538+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:14.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:14.538+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:15.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:07:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:07:15.209+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12512 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:07:15.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:15.207+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:15.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:15.207+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:15.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:15.207+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:15.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:15.207+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:15.800 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:15.532+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:15.801 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:15.532+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:15.801 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:15.532+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:15.801 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:15.533+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:16.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:15.803+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:16.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:15.803+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:16.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:15.803+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:16.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:15.804+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:16.555 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:16.230+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:16.556 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:16.230+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:16.556 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:16.230+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:16.556 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:16.231+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:16.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:16.772+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:16.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:16.772+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:16.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:16.772+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:16.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:16.773+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:16.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:16.558+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:16.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:16.558+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:16.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:16.558+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:16.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:16.558+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:17.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:17.210+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:17.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:17.210+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:17.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:17.210+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:17.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:17.211+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:17.819 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:17.569+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:17.819 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:17.569+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:17.820 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:17.569+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:17.820 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:17.570+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:18.180 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:17.822+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:18.181 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:17.822+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:18.181 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:17.822+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:18.181 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:17.823+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:18.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:18.183+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:18.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:18.183+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:18.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:18.183+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:18.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:18.184+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:18.846 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:18.577+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:18.846 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:18.577+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:18.846 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:18.577+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:18.846 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:18.577+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:19.168 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:18.848+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:19.169 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:18.848+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:19.169 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:18.848+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:19.169 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:18.849+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:19.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:19.171+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:19.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:19.171+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:19.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:19.171+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:19.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:19.172+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:19.875 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:19.555+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:19.875 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:19.555+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:19.875 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:19.555+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:19.875 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:19.556+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:20.136 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:19.877+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:20.136 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:19.877+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:20.137 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:19.877+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:20.137 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:19.878+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:20.452 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:07:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:07:20.210+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12607 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:07:20.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:20.139+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:20.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:20.139+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:20.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:20.139+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:20.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:20.139+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:20.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:20.524+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:20.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:20.524+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:20.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:20.524+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:20.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:20.524+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:21.178 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:20.909+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:21.178 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:20.909+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:21.178 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:20.909+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:21.178 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:20.910+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:21.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:21.180+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:21.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:21.180+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:21.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:21.180+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:21.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:21.181+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:21.862 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:21.529+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:21.862 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:21.529+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:21.862 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:21.529+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:21.863 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:21.529+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:22.215 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:21.864+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:22.215 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:21.864+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:22.215 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:21.864+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:22.215 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:21.865+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:22.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:22.217+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:22.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:22.217+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:22.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:22.217+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:22.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:22.218+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:22.875 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:22.510+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:22.875 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:22.510+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:22.876 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:22.510+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:22.876 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:22.511+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:23.167 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:22.878+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:23.167 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:22.878+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:23.167 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:22.878+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:23.167 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:22.879+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:23.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:23.169+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:23.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:23.169+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:23.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:23.169+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:23.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:23.170+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:23.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:23.528+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:23.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:23.528+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:23.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:23.528+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:23.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:23.529+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:24.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:24.134+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:24.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:24.134+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:24.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:24.134+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:24.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:24.135+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:24.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:23.892+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:24.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:23.892+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:24.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:23.892+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:24.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:23.892+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:24.911 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:24.509+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:24.911 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:24.509+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:24.911 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:24.509+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:24.911 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:24.509+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:25.209 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:24.913+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:25.209 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:24.913+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:25.209 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:24.913+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:25.210 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:24.914+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:25.210 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:25.153+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:25.210 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:25.153+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:25.210 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:25.153+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:25.210 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:25.154+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:25.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:07:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:07:25.211+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12702 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:07:25.916 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:25.552+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:25.916 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:25.552+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:25.916 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:25.552+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:25.916 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:25.553+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:26.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:26.148+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:26.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:26.148+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:26.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:26.148+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:26.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:26.149+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:26.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:25.918+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:26.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:25.918+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:26.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:25.918+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:26.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:25.920+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:26.906 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:26.569+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:26.906 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:26.569+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:26.906 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:26.569+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:26.906 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:26.569+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:27.164 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:26.908+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:27.164 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:26.908+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:27.164 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:26.908+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:27.164 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:26.909+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:27.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:27.166+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:27.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:27.166+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:27.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:27.166+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:27.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:27.167+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:27.907 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:27.558+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:27.907 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:27.558+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:27.907 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:27.558+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:27.907 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:27.558+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:28.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:28.147+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:28.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:28.147+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:28.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:28.147+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:28.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:28.148+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:28.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:27.909+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:28.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:27.909+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:28.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:27.909+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:28.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:27.910+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:28.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-01-23T09:07:28.758+0000 7fa9ee1d1700 0 monclient(hunting): authenticate timed out after 300 2024-01-23T09:07:28.757 INFO:teuthology.orchestra.run.smithi059.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-01-23T09:07:28.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:28.562+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:28.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:28.562+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:28.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:28.562+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:28.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:28.563+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:29.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:29.196+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:29.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:29.196+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:29.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:29.196+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:29.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:29.197+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:29.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:28.954+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:29.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:28.954+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:29.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:28.954+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:29.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:28.954+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:29.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-23T09:07:29.644 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/cephadm.py", line 1402, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/cephadm.py", line 40, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi059 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-01-23T09:07:29.859 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=cab3e883645d4de3bc39e5fd1350d988 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/cephadm.py", line 1402, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/cephadm.py", line 40, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi059 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-01-23T09:07:29.862 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-01-23T09:07:29.911 INFO:tasks.kclient:Unmounting kernel clients... 2024-01-23T09:07:29.912 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-23T09:07:29.912 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-23T09:07:29.943 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-23T09:07:29.943 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-23T09:07:29.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:29.523+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:29.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:29.523+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:29.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:29.523+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:29.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:29.524+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:29.986 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-23T09:07:30.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:07:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:07:30.213+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12796 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:07:30.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:30.168+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:30.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:30.168+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:30.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:30.168+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:30.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:30.169+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:30.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:29.972+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:30.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:29.972+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:30.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:29.972+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:30.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:29.973+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:30.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:30.527+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:30.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:30.527+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:30.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:30.527+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:30.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:30.528+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:31.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:31.124+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:31.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:31.124+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:31.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:31.124+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:31.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:31.124+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:31.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:30.979+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:31.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:30.979+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:31.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:30.979+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:31.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:30.979+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:31.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:31.560+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:31.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:31.560+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:31.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:31.560+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:31.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:31.561+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:32.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:32.096+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:32.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:32.096+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:32.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:32.096+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:32.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:32.097+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:32.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:31.976+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:32.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:31.976+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:32.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:31.976+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:32.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:31.977+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:32.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:32.543+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:32.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:32.543+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:32.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:32.543+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:32.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:32.544+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:33.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:33.066+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:33.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:33.066+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:33.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:33.066+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:33.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:33.067+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:33.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:32.993+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:33.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:32.993+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:33.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:32.993+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:33.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:32.995+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:33.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:33.576+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:33.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:33.576+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:33.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:33.576+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:33.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:33.576+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:34.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:34.022+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:34.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:34.022+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:34.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:34.022+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:34.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:34.023+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:34.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:34.044+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:34.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:34.044+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:34.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:34.044+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:34.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:34.045+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:34.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:34.615+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:34.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:34.615+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:34.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:34.615+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:34.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:34.615+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:35.332 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:07:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:07:35.214+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12892 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:07:35.333 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:34.999+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:35.333 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:34.999+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:35.333 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:34.999+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:35.333 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:34.999+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:35.333 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:34.998+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:35.334 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:34.998+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:35.334 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:34.998+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:35.334 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:34.999+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:35.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:35.576+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:35.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:35.576+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:35.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:35.576+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:35.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:35.577+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:36.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:35.973+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:36.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:35.974+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:36.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:35.974+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:36.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:35.974+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:36.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:35.996+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:36.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:35.996+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:36.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:35.996+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:36.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:35.997+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:36.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:36.585+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:36.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:36.585+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:36.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:36.585+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:36.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:36.586+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:37.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:36.954+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:37.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:36.954+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:37.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:36.954+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:37.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:36.955+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:37.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:37.023+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:37.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:37.023+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:37.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:37.023+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:37.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:37.023+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:37.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:37.540+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:37.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:37.540+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:37.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:37.540+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:37.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:37.541+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:38.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:38.071+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:38.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:38.071+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:38.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:38.071+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:38.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:38.072+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:38.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:38.005+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:38.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:38.005+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:38.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:38.005+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:38.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:38.006+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:38.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:38.587+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:38.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:38.587+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:38.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:38.587+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:38.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:38.588+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:39.243 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:39.045+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:39.243 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:39.045+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:39.243 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:39.045+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:39.243 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:39.046+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:39.244 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:38.992+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:39.244 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:38.992+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:39.244 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:38.992+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:39.244 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:38.992+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:39.908 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:39.550+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:39.909 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:39.550+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:39.909 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:39.550+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:39.909 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:39.551+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:40.213 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:40.078+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:40.213 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:40.078+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:40.213 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:40.078+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:40.214 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:40.079+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:40.214 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:39.958+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:40.214 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:39.958+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:40.214 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:39.958+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:40.214 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:39.959+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:40.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:07:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:07:40.215+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12987 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:07:40.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:40.583+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:40.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:40.583+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:40.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:40.583+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:40.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:40.584+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:41.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:41.086+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:41.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:41.086+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:41.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:41.086+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:41.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:41.086+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:41.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:40.999+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:41.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:40.999+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:41.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:40.999+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:41.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:41.000+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:41.956 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:41.555+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:41.956 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:41.556+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:41.956 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:41.556+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:41.956 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:41.556+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:42.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:42.064+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:42.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:42.064+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:42.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:42.064+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:42.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:42.064+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:42.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:42.017+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:42.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:42.017+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:42.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:42.017+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:42.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:42.018+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:42.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:42.574+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:42.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:42.574+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:42.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:42.574+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:42.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:42.575+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:43.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:43.108+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:43.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:43.108+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:43.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:43.108+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:43.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:43.109+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:43.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:43.046+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:43.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:43.046+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:43.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:43.046+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:43.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:43.047+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:43.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:43.579+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:43.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:43.579+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:43.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:43.579+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:43.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:43.580+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:44.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:44.157+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:44.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:44.157+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:44.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:44.157+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:44.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:44.158+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:44.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:44.011+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:44.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:44.011+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:44.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:44.011+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:44.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:44.011+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:44.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:44.566+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:44.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:44.566+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:44.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:44.566+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:44.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:44.567+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:45.239 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:07:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:07:45.217+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13082 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:07:45.240 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:44.987+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:45.240 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:44.987+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:45.240 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:44.987+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:45.240 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:44.988+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:45.241 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:45.191+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:45.241 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:45.191+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:45.241 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:45.191+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:45.241 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:45.192+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:45.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:45.526+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:45.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:45.526+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:45.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:45.526+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:45.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:45.527+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:46.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:46.212+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:46.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:46.212+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:46.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:46.212+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:46.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:46.213+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:46.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:45.978+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:46.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:45.978+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:46.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:45.978+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:46.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:45.979+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:46.928 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:46.522+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:46.928 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:46.522+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:46.928 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:46.522+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:46.928 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:46.522+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:47.182 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:46.930+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:47.182 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:46.930+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:47.183 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:46.930+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:47.183 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:46.931+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:47.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:47.184+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:47.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:47.184+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:47.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:47.184+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:47.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:47.185+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:47.905 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:47.501+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:47.906 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:47.501+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:47.906 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:47.501+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:47.906 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:47.502+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:48.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:47.908+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:48.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:47.908+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:48.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:47.908+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:48.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:47.908+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:48.462 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:48.172+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:48.462 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:48.172+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:48.462 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:48.172+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:48.462 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:48.173+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:48.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:48.464+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:48.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:48.464+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:48.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:48.464+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:48.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:48.465+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:49.165 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:48.880+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:49.166 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:48.880+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:49.166 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:48.880+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:49.166 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:48.881+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:49.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:49.168+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:49.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:49.168+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:49.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:49.168+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:49.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:49.168+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:49.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:49.450+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:49.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:49.450+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:49.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:49.450+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:49.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:49.451+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:50.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:49.923+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:50.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:49.923+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:50.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:49.923+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:50.198 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:49.924+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:50.456 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:07:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:07:50.218+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13176 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:07:50.457 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:50.199+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:50.457 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:50.199+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:50.457 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:50.199+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:50.457 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:50.200+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:50.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:50.459+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:50.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:50.459+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:50.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:50.459+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:50.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:50.459+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:51.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:50.945+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:51.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:50.945+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:51.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:50.945+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:51.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:50.945+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:51.679 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:51.241+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:51.679 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:51.241+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:51.679 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:51.241+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:51.680 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:51.242+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:51.680 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:51.439+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:51.680 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:51.439+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:51.680 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:51.439+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:51.680 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:51.440+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:52.273 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:51.975+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:52.274 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:51.975+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:52.274 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:51.975+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:52.274 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:51.976+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:52.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:52.275+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:52.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:52.275+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:52.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:52.275+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:52.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:52.276+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:52.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:52.466+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:52.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:52.467+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:52.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:52.467+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:52.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:52.467+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:53.320 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:52.989+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:53.320 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:52.989+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:53.321 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:52.989+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:53.321 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:52.990+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:53.721 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:53.322+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:53.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:53.322+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:53.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:53.322+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:53.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:53.324+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:53.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:53.475+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:53.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:53.475+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:53.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:53.475+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:53.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:53.476+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:54.244 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:53.986+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:54.244 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:53.986+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:54.244 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:53.986+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:54.244 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:53.987+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:54.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:54.347+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:54.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:54.347+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:54.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:54.347+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:54.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:54.347+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:54.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:54.473+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:54.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:54.473+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:54.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:54.473+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:54.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:54.474+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:55.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:07:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:07:55.219+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13273 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:07:55.298 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:55.005+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:55.298 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:55.005+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:55.298 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:55.005+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:55.298 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:55.006+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:55.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:55.300+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:55.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:55.300+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:55.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:55.300+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:55.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:55.301+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:55.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:55.463+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:55.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:55.463+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:55.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:55.463+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:55.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:55.464+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:56.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:55.999+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:56.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:55.999+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:56.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:55.999+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:56.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:56.000+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:56.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:56.315+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:56.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:56.315+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:56.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:56.315+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:56.477 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:56.316+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:56.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:56.478+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:56.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:56.478+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:56.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:56.478+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:56.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:56.479+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:57.291 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:57.036+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:57.292 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:57.036+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:57.292 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:57.036+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:57.292 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:57.037+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:57.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:57.293+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:57.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:57.293+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:57.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:57.293+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:57.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:57.294+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:57.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:57.495+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:57.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:57.495+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:57.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:57.495+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:57.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:57.496+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:58.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:57.995+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:58.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:57.995+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:58.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:57.995+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:58.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:57.996+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:58.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:58.328+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:58.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:58.328+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:58.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:58.328+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:58.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:58.329+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:07:58.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:58.527+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:58.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:58.527+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:58.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:58.527+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:58.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:58.528+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:59.359 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:58.996+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:59.359 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:58.996+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:59.360 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:58.996+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:07:59.360 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:58.997+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:59.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:59.510+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:59.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:59.510+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:59.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:59.510+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:07:59.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:07:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:07:59.511+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:07:59.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:59.360+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:59.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:59.360+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:59.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:59.360+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:07:59.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:07:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:07:59.361+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:00.228 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:08:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:08:00.219+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13368 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:08:00.228 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:59.978+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:00.228 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:59.978+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:00.229 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:59.978+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:00.229 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:07:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:07:59.979+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:00.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:00.317+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:00.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:00.317+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:00.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:00.317+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:00.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:00.318+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:00.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:00.469+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:00.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:00.469+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:00.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:00.469+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:00.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:00.470+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:01.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:00.961+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:01.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:00.961+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:01.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:00.961+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:01.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:00.961+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:01.680 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:01.366+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:01.680 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:01.366+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:01.680 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:01.367+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:01.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:01.368+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:01.681 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:01.430+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:01.681 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:01.430+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:01.681 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:01.430+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:01.681 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:01.430+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:02.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:01.951+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:02.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:01.951+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:02.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:01.951+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:02.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:01.952+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:02.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:02.339+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:02.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:02.339+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:02.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:02.339+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:02.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:02.340+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:02.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:02.402+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:02.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:02.402+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:02.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:02.402+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:02.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:02.403+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:03.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:02.947+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:03.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:02.947+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:03.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:02.947+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:03.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:02.948+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:03.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:03.353+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:03.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:03.353+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:03.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:03.353+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:03.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:03.354+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:03.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:03.420+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:03.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:03.420+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:03.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:03.420+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:03.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:03.421+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:04.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:03.906+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:04.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:03.906+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:04.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:03.906+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:04.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:03.907+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:04.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:04.387+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:04.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:04.387+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:04.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:04.387+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:04.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:04.388+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:04.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:04.372+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:04.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:04.372+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:04.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:04.372+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:04.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:04.373+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:05.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:04.882+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:05.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:04.882+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:05.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:04.882+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:05.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:04.883+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:05.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:08:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:08:05.221+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13463 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:08:05.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:05.391+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:05.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:05.392+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:05.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:05.392+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:05.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:05.392+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:05.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:05.366+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:05.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:05.366+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:05.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:05.366+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:05.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:05.367+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:06.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:05.932+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:06.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:05.932+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:06.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:05.932+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:06.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:05.932+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:06.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:06.402+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:06.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:06.402+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:06.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:06.402+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:06.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:06.403+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:06.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:06.365+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:06.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:06.365+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:06.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:06.365+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:06.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:06.366+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:07.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:06.914+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:07.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:06.914+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:07.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:06.914+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:07.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:06.914+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:07.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:07.372+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:07.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:07.372+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:07.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:07.372+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:07.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:07.373+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:07.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:07.374+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:07.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:07.374+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:07.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:07.374+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:07.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:07.375+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:08.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:07.895+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:08.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:07.895+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:08.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:07.895+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:08.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:07.896+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:08.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:08.398+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:08.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:08.398+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:08.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:08.398+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:08.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:08.399+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:08.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:08.327+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:08.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:08.327+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:08.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:08.327+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:08.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:08.328+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:09.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:08.916+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:09.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:08.916+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:09.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:08.916+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:09.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:08.917+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:09.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:09.391+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:09.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:09.391+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:09.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:09.391+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:09.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:09.392+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:09.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:09.313+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:09.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:09.313+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:09.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:09.313+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:09.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:09.314+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:10.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:09.953+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:10.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:09.953+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:10.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:09.953+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:10.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:09.954+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:10.471 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:08:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:08:10.222+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13555 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:08:10.497 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:10.349+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:10.497 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:10.349+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:10.497 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:10.349+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:10.497 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:10.350+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:10.497 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:10.342+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:10.498 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:10.342+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:10.498 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:10.342+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:10.498 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:10.343+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:11.318 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:10.986+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:11.318 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:10.986+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:11.318 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:10.986+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:11.318 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:10.987+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:11.654 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:11.398+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:11.654 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:11.398+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:11.654 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:11.398+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:11.654 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:11.399+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:11.655 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:11.318+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:11.655 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:11.318+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:11.655 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:11.318+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:11.655 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:11.320+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:12.357 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:11.974+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:12.357 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:11.974+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:12.357 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:11.974+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:12.357 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:11.975+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:12.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:12.397+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:12.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:12.397+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:12.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:12.397+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:12.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:12.398+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:12.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:12.358+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:12.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:12.358+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:12.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:12.358+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:12.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:12.358+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:13.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:12.931+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:13.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:12.931+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:13.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:12.931+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:13.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:12.932+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:13.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:13.384+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:13.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:13.384+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:13.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:13.384+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:13.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:13.384+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:13.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:13.394+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:13.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:13.394+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:13.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:13.394+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:13.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:13.395+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:14.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:13.938+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:14.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:13.938+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:14.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:13.938+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:14.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:13.938+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:14.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:14.421+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:14.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:14.421+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:14.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:14.421+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:14.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:14.421+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:14.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:14.370+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:14.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:14.370+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:14.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:14.370+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:14.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:14.371+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:15.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:14.961+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:15.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:14.962+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:15.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:14.962+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:15.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:14.962+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:15.721 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:08:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:08:15.224+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13652 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:08:15.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:15.432+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:15.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:15.432+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:15.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:15.432+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:15.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:15.433+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:15.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:15.355+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:15.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:15.355+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:15.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:15.355+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:15.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:15.356+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:16.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:15.967+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:16.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:15.967+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:16.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:15.967+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:16.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:15.967+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:16.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:16.459+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:16.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:16.459+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:16.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:16.459+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:16.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:16.459+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:16.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:16.356+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:16.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:16.356+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:16.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:16.356+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:16.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:16.357+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:17.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:16.958+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:17.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:16.958+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:17.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:16.958+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:17.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:16.959+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:17.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:17.364+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:17.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:17.364+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:17.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:17.364+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:17.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:17.365+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:17.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:17.449+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:17.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:17.449+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:17.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:17.449+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:17.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:17.449+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:18.396 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:17.981+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:18.396 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:17.981+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:18.397 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:17.981+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:18.397 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:17.982+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:18.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:18.473+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:18.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:18.473+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:18.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:18.473+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:18.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:18.474+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:18.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:18.398+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:18.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:18.398+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:18.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:18.398+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:18.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:18.399+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:19.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:18.936+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:19.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:18.936+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:19.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:18.936+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:19.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:18.936+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:19.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:19.394+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:19.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:19.394+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:19.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:19.394+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:19.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:19.395+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:19.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:19.455+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:19.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:19.455+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:19.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:19.455+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:19.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:19.456+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:20.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:19.907+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:20.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:19.907+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:20.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:19.907+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:20.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:19.908+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:20.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:08:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:08:20.226+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13747 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:08:20.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:20.420+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:20.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:20.420+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:20.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:20.420+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:20.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:20.421+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:20.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:20.374+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:20.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:20.374+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:20.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:20.374+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:20.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:20.375+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:21.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:20.859+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:21.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:20.859+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:21.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:20.859+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:21.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:20.859+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:21.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:21.401+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:21.682 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:21.401+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:21.682 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:21.401+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:21.682 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:21.402+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:21.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:21.462+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:21.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:21.462+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:21.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:21.462+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:21.683 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:21.463+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:21.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:21.830+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:21.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:21.830+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:21.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:21.830+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:21.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:21.830+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:22.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:22.423+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:22.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:22.423+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:22.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:22.423+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:22.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:22.423+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:22.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:22.455+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:22.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:22.455+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:22.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:22.455+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:22.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:22.455+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:23.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:22.806+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:23.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:22.806+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:23.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:22.806+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:23.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:22.806+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:23.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:23.500+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:23.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:23.500+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:23.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:23.500+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:23.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:23.501+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:23.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:23.402+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:23.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:23.402+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:23.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:23.402+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:23.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:23.403+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:24.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:23.841+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:24.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:23.841+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:24.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:23.841+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:24.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:23.842+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:24.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:24.534+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:24.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:24.534+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:24.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:24.534+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:24.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:24.535+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:24.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:24.363+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:24.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:24.363+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:24.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:24.363+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:24.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:24.363+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:25.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:24.841+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:25.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:24.841+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:25.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:24.841+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:25.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:24.842+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:25.524 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:08:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:08:25.227+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13843 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:08:25.524 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:25.325+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:25.524 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:25.325+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:25.524 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:25.325+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:25.525 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:25.326+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:25.818 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:25.526+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:25.819 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:25.526+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:25.819 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:25.526+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:25.819 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:25.526+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:26.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:25.820+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:26.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:25.820+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:26.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:25.820+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:26.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:25.821+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:26.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:26.322+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:26.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:26.322+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:26.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:26.322+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:26.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:26.323+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:26.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:26.499+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:26.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:26.499+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:26.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:26.499+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:26.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:26.500+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:27.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:26.857+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:27.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:26.857+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:27.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:26.857+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:27.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:26.858+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:27.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:27.298+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:27.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:27.298+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:27.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:27.298+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:27.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:27.299+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:27.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:27.509+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:27.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:27.509+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:27.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:27.509+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:27.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:27.509+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:28.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:27.831+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:28.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:27.831+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:28.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:27.831+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:28.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:27.832+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:28.515 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:28.251+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:28.515 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:28.251+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:28.516 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:28.251+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:28.516 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:28.251+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:28.810 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:28.517+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:28.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:28.517+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:28.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:28.517+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:28.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:28.518+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:29.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:28.812+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:29.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:28.812+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:29.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:28.812+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:29.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:28.813+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:29.540 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:29.280+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:29.540 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:29.280+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:29.540 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:29.280+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:29.540 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:29.281+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:29.801 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:29.542+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:29.801 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:29.542+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:29.801 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:29.542+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:29.801 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:29.543+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:30.149 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:29.803+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:30.149 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:29.803+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:30.149 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:29.803+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:30.149 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:29.803+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:30.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:08:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:08:30.229+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13938 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:08:30.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:30.312+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:30.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:30.312+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:30.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:30.312+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:30.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:30.313+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:30.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:30.531+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:30.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:30.531+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:30.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:30.531+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:30.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:30.531+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:30.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:30.769+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:30.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:30.769+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:30.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:30.769+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:30.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:30.769+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:31.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:31.312+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:31.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:31.312+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:31.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:31.312+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:31.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:31.313+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:31.775 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:31.507+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:31.775 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:31.507+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:31.775 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:31.507+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:31.776 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:31.508+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:32.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:31.777+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:32.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:31.777+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:32.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:31.777+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:32.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:31.778+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:32.527 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:32.279+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:32.528 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:32.279+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:32.528 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:32.279+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:32.528 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:32.280+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:32.528 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:32.529+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:32.528 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:32.529+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:32.529 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:32.529+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:32.529 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:32.529+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:33.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:32.788+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:33.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:32.788+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:33.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:32.788+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:33.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:32.789+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:33.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:33.328+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:33.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:33.328+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:33.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:33.328+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:33.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:33.329+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:33.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:33.554+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:33.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:33.554+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:33.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:33.554+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:33.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:33.555+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:34.180 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:33.831+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:34.180 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:33.831+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:34.180 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:33.831+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:34.180 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:33.831+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:34.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:34.326+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:34.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:34.326+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:34.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:34.326+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:34.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:34.326+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:34.856 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:34.595+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:34.856 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:34.595+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:34.856 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:34.595+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:34.856 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:34.596+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:35.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:34.858+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:35.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:34.858+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:35.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:34.858+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:35.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:34.858+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:35.643 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:08:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:08:35.230+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14035 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:08:35.644 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:35.337+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:35.644 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:35.337+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:35.644 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:35.337+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:35.644 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:35.338+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:35.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:35.849+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:35.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:35.849+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:35.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:35.849+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:35.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:35.849+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:35.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:35.645+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:35.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:35.645+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:35.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:35.645+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:35.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:35.646+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:36.612 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:36.303+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:36.612 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:36.303+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:36.612 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:36.303+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:36.613 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:36.303+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:36.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:36.844+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:36.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:36.844+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:36.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:36.844+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:36.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:36.845+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:36.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:36.614+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:36.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:36.614+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:36.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:36.614+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:36.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:36.615+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:37.627 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:37.254+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:37.627 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:37.254+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:37.627 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:37.254+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:37.627 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:37.255+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:37.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:37.815+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:37.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:37.815+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:37.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:37.815+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:37.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:37.816+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:37.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:37.629+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:37.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:37.629+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:37.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:37.629+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:37.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:37.630+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:38.611 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:38.283+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:38.611 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:38.284+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:38.611 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:38.284+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:38.611 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:38.284+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:38.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:38.781+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:38.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:38.781+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:38.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:38.781+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:38.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:38.782+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:38.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:38.612+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:38.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:38.612+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:38.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:38.612+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:38.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:38.613+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:39.607 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:39.324+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:39.607 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:39.324+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:39.607 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:39.324+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:39.607 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:39.325+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:39.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:39.757+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:39.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:39.757+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:39.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:39.757+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:39.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:39.758+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:39.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:39.608+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:39.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:39.608+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:39.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:39.608+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:39.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:39.609+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:40.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:08:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:08:40.232+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14131 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:08:40.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:40.335+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:40.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:40.335+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:40.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:40.335+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:40.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:40.336+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:40.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:40.744+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:40.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:40.744+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:40.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:40.744+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:40.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:40.745+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:40.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:40.571+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:40.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:40.571+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:40.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:40.571+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:40.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:40.571+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:41.684 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:41.301+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:41.684 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:41.301+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:41.685 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:41.301+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:41.685 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:41.302+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:41.685 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:41.550+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:41.685 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:41.550+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:41.685 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:41.550+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:41.685 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:41.551+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:41.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:41.703+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:41.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:41.703+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:41.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:41.703+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:41.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:41.704+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:42.668 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:42.344+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:42.669 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:42.344+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:42.669 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:42.344+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:42.669 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:42.345+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:42.669 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:42.564+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:42.669 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:42.564+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:42.670 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:42.564+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:42.670 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:42.564+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:42.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:42.670+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:42.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:42.670+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:42.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:42.670+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:42.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:42.671+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:43.655 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:43.347+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:43.655 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:43.347+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:43.655 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:43.347+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:43.655 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:43.347+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:43.656 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:43.552+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:43.656 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:43.552+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:43.656 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:43.552+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:43.656 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:43.553+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:43.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:43.656+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:43.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:43.656+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:43.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:43.656+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:43.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:43.657+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:44.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:44.529+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:44.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:44.529+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:44.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:44.529+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:44.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:44.530+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:44.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:44.632+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:44.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:44.632+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:44.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:44.632+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:44.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:44.633+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:44.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:44.392+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:44.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:44.392+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:44.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:44.392+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:44.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:44.393+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:45.518 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:08:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:08:45.233+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14227 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:08:45.518 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:45.423+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:45.519 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:45.423+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:45.519 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:45.423+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:45.519 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:45.424+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:45.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:45.642+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:45.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:45.642+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:45.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:45.642+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:45.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:45.643+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:45.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:45.520+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:45.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:45.520+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:45.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:45.520+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:45.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:45.521+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:46.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:46.489+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:46.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:46.489+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:46.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:46.489+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:46.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:46.490+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:46.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:46.599+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:46.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:46.599+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:46.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:46.599+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:46.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:46.601+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:46.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:46.411+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:46.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:46.411+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:46.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:46.411+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:46.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:46.412+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:47.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:47.457+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:47.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:47.457+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:47.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:47.457+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:47.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:47.457+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:47.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:47.569+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:47.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:47.569+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:47.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:47.569+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:47.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:47.569+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:47.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:47.407+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:47.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:47.407+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:47.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:47.407+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:47.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:47.408+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:48.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:48.450+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:48.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:48.450+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:48.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:48.450+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:48.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:48.450+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:48.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:48.574+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:48.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:48.574+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:48.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:48.574+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:48.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:48.575+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:48.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:48.479+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:48.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:48.479+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:48.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:48.479+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:48.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:48.480+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:49.526 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:49.473+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:49.527 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:49.473+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:49.527 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:49.473+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:49.527 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:49.474+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:49.527 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:49.453+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:49.527 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:49.453+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:49.527 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:49.453+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:49.528 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:49.453+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:49.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:49.528+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:49.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:49.528+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:49.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:49.528+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:49.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:49.529+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:50.433 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:08:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:08:50.234+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14322 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:08:50.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:50.435+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:50.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:50.435+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:50.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:50.435+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:50.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:50.436+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:50.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:50.447+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:50.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:50.447+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:50.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:50.447+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:50.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:50.448+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:50.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:50.530+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:50.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:50.530+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:50.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:50.530+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:50.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:50.531+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:51.685 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:51.566+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:51.686 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:51.566+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:51.686 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:51.566+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:51.686 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:51.567+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:51.686 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:51.414+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:51.687 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:51.414+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:51.687 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:51.414+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:51.687 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:51.415+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:51.687 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:51.420+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:51.688 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:51.420+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:51.688 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:51.420+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:51.688 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:51.421+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:52.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:52.557+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:52.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:52.557+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:52.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:52.557+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:52.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:52.558+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:52.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:52.378+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:52.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:52.378+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:52.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:52.378+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:52.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:52.379+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:52.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:52.443+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:52.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:52.443+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:52.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:52.443+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:52.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:52.444+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:53.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:53.401+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:53.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:53.401+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:53.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:53.401+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:53.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:53.402+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:53.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:53.566+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:53.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:53.566+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:53.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:53.566+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:53.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:53.567+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:53.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:53.363+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:53.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:53.363+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:53.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:53.363+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:53.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:53.363+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:54.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:54.561+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:54.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:54.561+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:54.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:54.561+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:54.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:54.562+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:54.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:54.381+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:54.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:54.381+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:54.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:54.381+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:54.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:54.381+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:54.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:54.353+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:54.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:54.353+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:54.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:54.353+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:54.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:54.354+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:55.533 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:08:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:08:55.236+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14412 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:08:55.533 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:55.385+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:55.533 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:55.385+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:55.533 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:55.385+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:55.534 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:55.386+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:55.534 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:55.380+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:55.534 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:55.380+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:55.534 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:55.380+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:55.534 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:55.380+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:55.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:55.534+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:55.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:55.534+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:55.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:55.534+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:55.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:55.535+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:56.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:56.548+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:56.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:56.548+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:56.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:56.548+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:56.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:56.549+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:56.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:56.336+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:56.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:56.336+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:56.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:56.336+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:56.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:56.337+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:56.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:56.345+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:56.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:56.345+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:56.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:56.345+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:56.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:56.345+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:57.571 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:57.291+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:57.572 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:57.291+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:57.572 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:57.291+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:57.572 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:57.291+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:57.572 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:57.307+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:57.572 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:57.307+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:57.573 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:57.307+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:57.573 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:57.308+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:57.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:57.573+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:57.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:57.573+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:57.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:57.573+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:57.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:57.574+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:58.596 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:58.351+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:58.596 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:58.351+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:58.597 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:58.351+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:58.597 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:58.353+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:58.597 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:58.330+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:58.597 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:58.330+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:58.597 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:58.330+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:58.597 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:58.331+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:58.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:58.598+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:58.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:58.598+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:58.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:58.598+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:58.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:58.599+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:59.561 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:59.374+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:59.561 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:59.374+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:59.562 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:59.374+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:08:59.562 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:08:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:08:59.375+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:08:59.562 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:59.318+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:59.562 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:59.318+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:59.562 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:59.318+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:08:59.562 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:08:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:08:59.318+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:08:59.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:59.563+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:59.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:59.563+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:59.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:59.563+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:08:59.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:08:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:08:59.564+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:00.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:09:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:09:00.237+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14495 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:09:00.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:00.304+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:00.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:00.304+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:00.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:00.304+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:00.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:00.305+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:00.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:00.327+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:00.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:00.327+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:00.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:00.327+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:00.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:00.328+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:00.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:00.547+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:00.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:00.547+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:00.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:00.547+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:00.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:00.548+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:01.533 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:01.375+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:01.534 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:01.375+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:01.534 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:01.375+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:01.534 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:01.375+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:01.534 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:01.279+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:01.534 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:01.279+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:01.534 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:01.279+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:01.534 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:01.280+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:01.829 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:01.535+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:01.829 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:01.535+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:01.829 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:01.535+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:01.829 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:01.536+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:02.552 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:02.418+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:02.553 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:02.418+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:02.553 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:02.418+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:02.553 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:02.419+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:02.553 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:02.292+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:02.553 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:02.292+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:02.553 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:02.292+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:02.554 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:02.293+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:02.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:02.554+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:02.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:02.554+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:02.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:02.554+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:02.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:02.555+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:03.563 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:03.422+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:03.564 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:03.422+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:03.564 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:03.422+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:03.564 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:03.423+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:03.564 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:03.282+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:03.564 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:03.282+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:03.564 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:03.282+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:03.564 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:03.283+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:03.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:03.577+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:03.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:03.577+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:03.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:03.577+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:03.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:03.578+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:04.585 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:04.400+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:04.585 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:04.400+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:04.586 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:04.400+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:04.586 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:04.401+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:04.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:04.261+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:04.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:04.261+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:04.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:04.261+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:04.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:04.261+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:04.900 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:04.587+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:04.900 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:04.587+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:04.901 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:04.587+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:04.901 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:04.588+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:05.586 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:09:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:09:05.239+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14579 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:09:05.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:05.277+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:05.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:05.277+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:05.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:05.277+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:05.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:05.277+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:05.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:05.364+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:05.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:05.364+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:05.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:05.364+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:05.587 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:05.365+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:05.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:05.587+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:05.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:05.587+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:05.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:05.587+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:05.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:05.588+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:06.541 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:06.334+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:06.541 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:06.334+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:06.542 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:06.334+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:06.542 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:06.335+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:06.542 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:06.282+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:06.542 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:06.282+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:06.542 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:06.282+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:06.543 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:06.283+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:06.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:06.543+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:06.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:06.543+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:06.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:06.543+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:06.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:06.544+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:07.574 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:07.364+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:07.574 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:07.364+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:07.574 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:07.364+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:07.574 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:07.364+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:07.575 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:07.286+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:07.575 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:07.286+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:07.575 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:07.286+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:07.575 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:07.287+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:07.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:07.575+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:07.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:07.575+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:07.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:07.575+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:07.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:07.576+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:08.602 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:08.321+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:08.602 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:08.321+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:08.602 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:08.321+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:08.603 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:08.322+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:08.603 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:08.303+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:08.603 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:08.303+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:08.603 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:08.303+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:08.603 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:08.304+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:08.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:08.604+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:08.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:08.604+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:08.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:08.604+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:08.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:08.605+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:09.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:09.372+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:09.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:09.372+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:09.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:09.372+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:09.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:09.373+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:09.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:09.338+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:09.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:09.338+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:09.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:09.338+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:09.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:09.339+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:09.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:09.590+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:09.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:09.590+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:09.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:09.590+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:09.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:09.591+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:10.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:09:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:09:10.240+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14662 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:09:10.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:10.333+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:10.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:10.333+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:10.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:10.333+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:10.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:10.334+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:10.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:10.356+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:10.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:10.356+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:10.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:10.356+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:10.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:10.357+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:10.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:10.604+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:10.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:10.604+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:10.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:10.604+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:10.977 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:10.605+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:11.609 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:11.365+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:11.609 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:11.365+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:11.609 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:11.365+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:11.609 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:11.366+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:11.609 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:11.348+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:11.610 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:11.348+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:11.610 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:11.348+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:11.610 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:11.349+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:11.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:11.610+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:11.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:11.610+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:11.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:11.610+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:11.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:11.611+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:12.596 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:12.328+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:12.597 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:12.328+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:12.597 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:12.328+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:12.597 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:12.329+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:12.597 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:12.368+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:12.597 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:12.368+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:12.597 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:12.368+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:12.598 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:12.369+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:12.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:12.598+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:12.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:12.598+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:12.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:12.598+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:12.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:12.599+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:13.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:13.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:13.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:13.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:13.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:13.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:13.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:13.369+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:13.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:13.572+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:13.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:13.572+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:13.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:13.572+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:13.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:13.572+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:13.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:13.376+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:13.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:13.376+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:13.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:13.376+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:13.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:13.377+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:14.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:14.401+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:14.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:14.401+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:14.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:14.401+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:14.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:14.402+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:14.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:14.525+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:14.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:14.525+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:14.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:14.525+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:14.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:14.526+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:14.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:14.417+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:14.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:14.417+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:14.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:14.418+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:14.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:14.418+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:15.507 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:09:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:09:15.242+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14746 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:09:15.507 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:15.389+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:15.507 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:15.389+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:15.507 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:15.389+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:15.507 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:15.390+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:15.508 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:15.451+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:15.508 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:15.451+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:15.508 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:15.451+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:15.508 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:15.451+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:15.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:15.508+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:15.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:15.508+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:15.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:15.508+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:15.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:15.509+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:16.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:16.427+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:16.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:16.427+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:16.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:16.427+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:16.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:16.428+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:16.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:16.547+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:16.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:16.547+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:16.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:16.547+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:16.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:16.548+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:16.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:16.374+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:16.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:16.374+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:16.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:16.374+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:16.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:16.374+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:17.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:17.404+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:17.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:17.404+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:17.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:17.404+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:17.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:17.405+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:17.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:17.412+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:17.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:17.412+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:17.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:17.412+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:17.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:17.413+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:17.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:17.566+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:17.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:17.566+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:17.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:17.566+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:17.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:17.567+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:18.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:18.407+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:18.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:18.407+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:18.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:18.407+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:18.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:18.408+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:18.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:18.527+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:18.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:18.527+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:18.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:18.527+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:18.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:18.528+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:18.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:18.406+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:18.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:18.406+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:18.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:18.406+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:18.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:18.407+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:19.538 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:19.373+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:19.539 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:19.373+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:19.539 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:19.373+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:19.539 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:19.373+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:19.539 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:19.447+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:19.539 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:19.447+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:19.540 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:19.447+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:19.540 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:19.447+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:19.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:19.539+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:19.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:19.539+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:19.972 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:19.539+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:19.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:19.540+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:20.420 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:09:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:09:20.242+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14828 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:09:20.420 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:20.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:20.420 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:20.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:20.420 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:20.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:20.420 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:20.369+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:20.548 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:20.420+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:20.548 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:20.420+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:20.548 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:20.420+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:20.548 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:20.422+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:20.548 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:20.547+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:20.549 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:20.547+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:20.549 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:20.547+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:20.549 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:20.548+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:21.687 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:21.522+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:21.687 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:21.522+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:21.688 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:21.522+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:21.688 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:21.523+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:21.688 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:21.405+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:21.688 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:21.405+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:21.688 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:21.405+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:21.689 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:21.406+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:21.689 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:21.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:21.689 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:21.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:21.689 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:21.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:21.689 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:21.376+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:22.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:22.502+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:22.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:22.502+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:22.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:22.502+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:22.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:22.503+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:22.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:22.442+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:22.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:22.442+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:22.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:22.442+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:22.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:22.443+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:22.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:22.425+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:22.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:22.425+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:22.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:22.425+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:22.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:22.426+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:23.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:23.394+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:23.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:23.394+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:23.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:23.394+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:23.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:23.394+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:23.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:23.438+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:23.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:23.438+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:23.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:23.438+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:23.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:23.439+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:23.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:23.486+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:23.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:23.486+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:23.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:23.486+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:23.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:23.487+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:24.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:24.477+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:24.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:24.477+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:24.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:24.477+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:24.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:24.477+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:24.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:24.470+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:24.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:24.470+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:24.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:24.470+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:24.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:24.471+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:24.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:24.427+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:24.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:24.427+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:24.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:24.427+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:24.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:24.428+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:25.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:09:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:09:25.244+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14911 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:09:25.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:25.466+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:25.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:25.466+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:25.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:25.466+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:25.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:25.466+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:25.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:25.461+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:25.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:25.461+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:25.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:25.461+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:25.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:25.462+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:25.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:25.432+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:25.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:25.432+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:25.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:25.432+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:25.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:25.433+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:26.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:26.485+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:26.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:26.485+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:26.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:26.485+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:26.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:26.486+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:26.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:26.465+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:26.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:26.465+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:26.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:26.465+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:26.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:26.466+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:26.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:26.407+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:26.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:26.407+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:26.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:26.407+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:26.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:26.408+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:27.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:27.443+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:27.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:27.443+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:27.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:27.443+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:27.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:27.444+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:27.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:27.402+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:27.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:27.402+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:27.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:27.402+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:27.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:27.402+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:27.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:27.436+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:27.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:27.436+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:27.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:27.436+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:27.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:27.437+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:28.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:28.400+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:28.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:28.400+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:28.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:28.400+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:28.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:28.400+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:28.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:28.359+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:28.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:28.359+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:28.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:28.359+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:28.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:28.360+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:28.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:28.393+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:28.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:28.393+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:28.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:28.393+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:28.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:28.394+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:29.477 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:29.433+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:29.477 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:29.433+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:29.477 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:29.433+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:29.477 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:29.434+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:29.478 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:29.397+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:29.478 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:29.397+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:29.478 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:29.397+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:29.478 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:29.398+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:29.479 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:29.345+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:29.479 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:29.345+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:29.479 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:29.345+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:29.479 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:29.346+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:30.060 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-23T09:09:30.061 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2024-01-23T09:09:30.061 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-23T09:09:30.061 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-23T09:09:30.078 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-01-23T09:09:30.079 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-23T09:09:30.135 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-23T09:09:30.343 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:09:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:09:30.245+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14992 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:09:30.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:30.438+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:30.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:30.438+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:30.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:30.438+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:30.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:30.438+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:30.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:30.343+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:30.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:30.343+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:30.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:30.343+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:30.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:30.344+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:30.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:30.438+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:30.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:30.438+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:30.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:30.438+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:30.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:30.438+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:31.688 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:31.485+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:31.688 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:31.485+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:31.689 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:31.485+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:31.689 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:31.486+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:31.689 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:31.389+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:31.689 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:31.389+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:31.689 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:31.389+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:31.689 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:31.390+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:31.690 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:31.423+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:31.690 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:31.423+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:31.690 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:31.423+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:31.690 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:31.424+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:32.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:32.345+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:32.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:32.345+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:32.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:32.345+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:32.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:32.346+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:32.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:32.522+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:32.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:32.522+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:32.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:32.522+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:32.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:32.523+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:32.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:32.439+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:32.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:32.439+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:32.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:32.439+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:32.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:32.439+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:33.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:33.374+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:33.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:33.374+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:33.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:33.374+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:33.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:33.375+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:33.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:33.520+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:33.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:33.520+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:33.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:33.520+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:33.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:33.520+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:33.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:33.467+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:33.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:33.467+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:33.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:33.467+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:33.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:33.467+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:34.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:34.393+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:34.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:34.393+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:34.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:34.393+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:34.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:34.393+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:34.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:34.508+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:34.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:34.508+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:34.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:34.508+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:34.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:34.508+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:34.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:34.450+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:34.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:34.450+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:34.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:34.450+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:34.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:34.451+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:35.511 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:09:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:09:35.247+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15076 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:09:35.511 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:35.374+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:35.511 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:35.374+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:35.511 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:35.374+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:35.512 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:35.374+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:35.512 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:35.401+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:35.512 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:35.401+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:35.512 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:35.401+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:35.512 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:35.402+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:35.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:35.511+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:35.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:35.511+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:35.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:35.511+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:35.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:35.512+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:36.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:36.386+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:36.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:36.386+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:36.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:36.386+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:36.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:36.386+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:36.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:36.468+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:36.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:36.468+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:36.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:36.468+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:36.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:36.468+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:36.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:36.386+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:36.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:36.386+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:36.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:36.386+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:36.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:36.387+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:37.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:37.350+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:37.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:37.350+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:37.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:37.350+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:37.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:37.351+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:37.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:37.482+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:37.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:37.482+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:37.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:37.482+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:37.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:37.482+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:37.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:37.393+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:37.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:37.393+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:37.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:37.393+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:37.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:37.394+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:38.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:38.345+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:38.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:38.345+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:38.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:38.345+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:38.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:38.346+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:38.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:38.526+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:38.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:38.526+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:38.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:38.526+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:38.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:38.527+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:38.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:38.381+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:38.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:38.381+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:38.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:38.381+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:38.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:38.382+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:39.525 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:39.362+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:39.525 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:39.362+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:39.525 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:39.362+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:39.526 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:39.362+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:39.526 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:39.417+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:39.526 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:39.417+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:39.526 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:39.417+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:39.526 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:39.418+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:39.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:39.526+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:39.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:39.526+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:39.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:39.526+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:39.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:39.527+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:40.419 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:09:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:09:40.250+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15156 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:09:40.419 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:40.334+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:40.419 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:40.334+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:40.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:40.334+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:40.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:40.335+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:40.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:40.480+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:40.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:40.480+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:40.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:40.480+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:40.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:40.480+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:40.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:40.420+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:40.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:40.420+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:40.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:40.420+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:40.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:40.421+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:41.689 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:41.501+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:41.689 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:41.501+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:41.689 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:41.501+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:41.690 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:41.502+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:41.690 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:41.332+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:41.690 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:41.332+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:41.690 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:41.332+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:41.690 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:41.333+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:41.691 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:41.419+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:41.691 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:41.419+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:41.691 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:41.419+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:41.691 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:41.420+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:42.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:42.304+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:42.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:42.304+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:42.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:42.304+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:42.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:42.304+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:42.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:42.481+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:42.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:42.481+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:42.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:42.481+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:42.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:42.482+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:42.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:42.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:42.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:42.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:42.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:42.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:42.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:42.377+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:43.567 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:43.465+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:43.567 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:43.465+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:43.567 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:43.465+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:43.568 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:43.466+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:43.568 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:43.279+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:43.568 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:43.279+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:43.568 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:43.279+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:43.568 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:43.279+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:43.569 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:43.365+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:43.569 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:43.365+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:43.569 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:43.365+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:43.569 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:43.366+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:44.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:44.292+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:44.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:44.292+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:44.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:44.292+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:44.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:44.292+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:44.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:44.348+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:44.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:44.348+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:44.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:44.348+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:44.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:44.349+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:44.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:44.492+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:44.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:44.492+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:44.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:44.492+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:44.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:44.493+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:45.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:09:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:09:45.251+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15238 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:09:45.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:45.326+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:45.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:45.326+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:45.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:45.326+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:45.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:45.326+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:45.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:45.365+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:45.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:45.365+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:45.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:45.365+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:45.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:45.366+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:45.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:45.466+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:45.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:45.466+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:45.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:45.466+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:45.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:45.467+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:46.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:46.312+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:46.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:46.312+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:46.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:46.312+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:46.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:46.313+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:46.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:46.364+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:46.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:46.364+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:46.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:46.364+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:46.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:46.365+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:46.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:46.517+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:46.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:46.517+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:46.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:46.517+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:46.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:46.517+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:47.623 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:47.278+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:47.624 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:47.278+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:47.624 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:47.278+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:47.624 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:47.279+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:47.624 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:47.519+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:47.624 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:47.519+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:47.625 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:47.519+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:47.625 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:47.520+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:47.625 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:47.375+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:47.625 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:47.375+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:47.625 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:47.375+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:47.625 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:47.376+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:48.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:48.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:48.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:48.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:48.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:48.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:48.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:48.376+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:48.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:48.257+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:48.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:48.257+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:48.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:48.257+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:48.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:48.258+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:48.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:48.471+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:48.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:48.471+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:48.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:48.471+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:48.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:48.472+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:49.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:49.400+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:49.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:49.400+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:49.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:49.400+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:49.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:49.401+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:49.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:49.237+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:49.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:49.237+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:49.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:49.237+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:49.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:49.237+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:49.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:49.469+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:49.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:49.469+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:49.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:49.469+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:49.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:49.470+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:50.426 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:09:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:09:50.252+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15318 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:09:50.426 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:50.245+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:50.426 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:50.245+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:50.426 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:50.245+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:50.426 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:50.246+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:50.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:50.437+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:50.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:50.437+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:50.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:50.437+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:50.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:50.438+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:50.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:50.427+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:50.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:50.427+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:50.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:50.427+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:50.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:50.428+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:51.440 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:51.211+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:51.440 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:51.211+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:51.441 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:51.211+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:51.441 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:51.212+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:51.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:51.444+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:51.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:51.444+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:51.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:51.444+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:51.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:51.444+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:51.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:51.442+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:51.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:51.442+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:51.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:51.442+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:51.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:51.442+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:52.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:52.254+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:52.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:52.254+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:52.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:52.254+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:52.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:52.255+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:52.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:52.445+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:52.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:52.445+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:52.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:52.445+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:52.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:52.446+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:52.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:52.401+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:52.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:52.401+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:52.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:52.401+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:52.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:52.402+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:53.568 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:53.227+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:53.568 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:53.227+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:53.568 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:53.227+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:53.568 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:53.227+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:53.569 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:53.421+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:53.569 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:53.421+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:53.569 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:53.421+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:53.569 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:53.422+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:53.569 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:53.446+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:53.570 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:53.446+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:53.570 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:53.446+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:53.570 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:53.446+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:54.417 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:54.273+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:54.417 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:54.273+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:54.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:54.273+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:54.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:54.274+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:54.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:54.468+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:54.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:54.468+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:54.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:54.468+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:54.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:54.469+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:54.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:54.419+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:54.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:54.419+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:54.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:54.419+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:54.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:54.420+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:55.722 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:09:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:09:55.254+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15402 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:09:55.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:55.224+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:55.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:55.224+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:55.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:55.224+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:55.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:55.225+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:55.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:55.468+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:55.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:55.468+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:55.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:55.468+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:55.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:55.469+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:55.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:55.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:55.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:55.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:55.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:55.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:55.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:55.376+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:56.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:56.186+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:56.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:56.186+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:56.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:56.186+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:56.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:56.187+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:56.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:56.421+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:56.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:56.421+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:56.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:56.421+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:56.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:56.421+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:56.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:56.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:56.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:56.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:56.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:56.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:56.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:56.369+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:57.440 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:57.382+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:57.440 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:57.382+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:57.440 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:57.382+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:57.440 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:57.382+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:57.441 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:57.190+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:57.441 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:57.190+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:57.441 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:57.190+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:57.441 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:57.191+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:57.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:57.441+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:57.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:57.442+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:57.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:57.442+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:57.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:57.443+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:58.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:58.373+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:58.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:58.373+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:58.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:58.373+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:58.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:58.373+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:58.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:58.240+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:58.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:58.240+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:58.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:58.240+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:58.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:58.241+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:58.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:58.411+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:58.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:58.411+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:58.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:58.411+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:58.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:58.412+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:59.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:59.455+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:59.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:59.455+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:59.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:59.455+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:09:59.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:09:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:09:59.455+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:09:59.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:59.251+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:59.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:59.251+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:59.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:59.251+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:09:59.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:09:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:09:59.251+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:09:59.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:59.362+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:59.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:59.362+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:59.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:59.362+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:09:59.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:09:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:09:59.362+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:00.424 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:10:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:10:00.256+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15483 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:10:00.425 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:00.247+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:00.425 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:00.247+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:00.425 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:00.247+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:00.425 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:00.248+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:00.425 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:00.341+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:00.426 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:00.341+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:00.426 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:00.341+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:00.426 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:00.342+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:00.709 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:00.426+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:00.709 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:00.426+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:00.709 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:00.426+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:00.710 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:00.427+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:01.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:01.401+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:01.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:01.402+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:01.472 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:01.402+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:01.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:01.402+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:01.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:01.228+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:01.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:01.228+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:01.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:01.228+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:01.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:01.229+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:01.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:01.298+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:01.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:01.298+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:01.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:01.298+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:01.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:01.298+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:02.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:02.403+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:02.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:02.403+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:02.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:02.403+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:02.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:02.403+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:02.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:02.252+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:02.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:02.252+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:02.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:02.252+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:02.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:02.253+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:02.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:02.306+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:02.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:02.306+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:02.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:02.306+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:02.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:02.307+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:03.569 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:03.240+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:03.569 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:03.240+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:03.569 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:03.240+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:03.569 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:03.240+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:03.570 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:03.279+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:03.570 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:03.279+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:03.570 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:03.279+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:03.570 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:03.280+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:03.570 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:03.427+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:03.570 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:03.427+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:03.571 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:03.427+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:03.571 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:03.427+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:04.458 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:04.209+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:04.459 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:04.209+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:04.459 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:04.209+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:04.459 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:04.210+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:04.459 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:04.308+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:04.459 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:04.308+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:04.460 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:04.308+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:04.460 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:04.308+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:04.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:04.460+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:04.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:04.460+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:04.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:04.460+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:04.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:04.461+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:05.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:10:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:10:05.257+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15568 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:10:05.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:05.273+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:05.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:05.273+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:05.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:05.273+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:05.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:05.273+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:05.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:05.184+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:05.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:05.184+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:05.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:05.184+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:05.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:05.185+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:05.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:05.479+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:05.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:05.479+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:05.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:05.479+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:05.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:05.480+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:06.455 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:06.298+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:06.455 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:06.298+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:06.456 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:06.298+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:06.456 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:06.299+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:06.456 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:06.163+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:06.456 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:06.163+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:06.456 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:06.163+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:06.457 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:06.164+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:06.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:06.457+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:06.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:06.457+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:06.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:06.457+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:06.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:06.458+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:07.457 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:07.156+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:07.457 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:07.156+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:07.457 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:07.156+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:07.458 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:07.157+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:07.458 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:07.344+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:07.458 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:07.344+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:07.458 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:07.344+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:07.458 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:07.344+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:07.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:07.458+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:07.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:07.458+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:07.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:07.458+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:07.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:07.459+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:08.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:08.163+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:08.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:08.163+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:08.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:08.163+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:08.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:08.164+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:08.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:08.375+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:08.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:08.375+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:08.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:08.375+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:08.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:08.376+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:08.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:08.502+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:08.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:08.502+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:08.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:08.502+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:08.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:08.502+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:09.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:09.177+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:09.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:09.177+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:09.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:09.177+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:09.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:09.178+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:09.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:09.396+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:09.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:09.396+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:09.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:09.396+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:09.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:09.397+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:09.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:09.523+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:09.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:09.523+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:09.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:09.523+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:09.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:09.524+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:10.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:10:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:10:10.259+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15650 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:10:10.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:10.224+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:10.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:10.224+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:10.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:10.224+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:10.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:10.225+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:10.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:10.406+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:10.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:10.406+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:10.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:10.406+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:10.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:10.407+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:10.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:10.500+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:10.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:10.500+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:10.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:10.500+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:10.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:10.500+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:11.691 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:11.240+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:11.691 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:11.240+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:11.692 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:11.240+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:11.692 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:11.241+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:11.692 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:11.486+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:11.692 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:11.486+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:11.692 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:11.486+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:11.693 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:11.487+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:11.693 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:11.437+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:11.693 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:11.437+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:11.693 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:11.437+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:11.693 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:11.437+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:12.486 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:12.475+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:12.486 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:12.475+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:12.486 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:12.475+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:12.486 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:12.475+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:12.486 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:12.236+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:12.487 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:12.236+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:12.487 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:12.236+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:12.487 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:12.237+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:12.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:12.526+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:12.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:12.526+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:12.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:12.526+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:12.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:12.527+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:13.490 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:13.232+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:13.490 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:13.232+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:13.490 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:13.232+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:13.491 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:13.233+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:13.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:13.572+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:13.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:13.572+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:13.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:13.572+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:13.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:13.573+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:13.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:13.492+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:13.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:13.492+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:13.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:13.492+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:13.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:13.493+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:14.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:14.195+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:14.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:14.195+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:14.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:14.195+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:14.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:14.195+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:14.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:14.564+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:14.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:14.564+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:14.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:14.564+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:14.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:14.565+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:14.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:14.488+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:14.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:14.488+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:14.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:14.488+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:14.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:14.488+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:15.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:10:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:10:15.260+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15734 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:10:15.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:15.196+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:15.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:15.196+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:15.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:15.196+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:15.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:15.197+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:15.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:15.478+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:15.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:15.478+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:15.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:15.478+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:15.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:15.479+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:15.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:15.566+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:15.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:15.566+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:15.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:15.566+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:15.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:15.566+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:16.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:16.185+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:16.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:16.185+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:16.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:16.185+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:16.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:16.186+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:16.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:16.494+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:16.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:16.494+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:16.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:16.494+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:16.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:16.495+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:16.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:16.546+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:16.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:16.546+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:16.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:16.546+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:16.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:16.547+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:17.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:17.208+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:17.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:17.208+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:17.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:17.208+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:17.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:17.208+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:17.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:17.477+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:17.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:17.477+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:17.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:17.477+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:17.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:17.478+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:17.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:17.572+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:17.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:17.572+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:17.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:17.572+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:17.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:17.573+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:18.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:18.206+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:18.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:18.206+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:18.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:18.206+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:18.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:18.207+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:18.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:18.488+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:18.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:18.488+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:18.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:18.488+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:18.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:18.489+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:18.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:18.590+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:18.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:18.590+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:18.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:18.590+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:18.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:18.591+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:19.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:19.164+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:19.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:19.164+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:19.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:19.164+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:19.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:19.165+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:19.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:19.483+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:19.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:19.483+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:19.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:19.483+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:19.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:19.483+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:19.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:19.602+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:19.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:19.602+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:19.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:19.602+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:19.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:19.602+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:20.451 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:10:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:10:20.262+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15816 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:10:20.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:20.158+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:20.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:20.158+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:20.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:20.158+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:20.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:20.158+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:20.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:20.453+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:20.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:20.453+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:20.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:20.453+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:20.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:20.453+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:20.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:20.643+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:20.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:20.643+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:20.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:20.643+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:20.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:20.644+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:21.454 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:21.189+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:21.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:21.189+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:21.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:21.189+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:21.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:21.189+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:21.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:21.456+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:21.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:21.456+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:21.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:21.456+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:21.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:21.456+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:21.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:21.681+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:21.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:21.681+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:21.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:21.681+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:21.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:21.682+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:22.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:22.191+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:22.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:22.191+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:22.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:22.191+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:22.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:22.192+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:22.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:22.724+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:22.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:22.724+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:22.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:22.724+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:22.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:22.724+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:22.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:22.506+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:22.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:22.506+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:22.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:22.506+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:22.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:22.507+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:23.464 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:23.207+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:23.464 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:23.207+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:23.464 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:23.207+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:23.465 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:23.208+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:23.465 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:23.465+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:23.465 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:23.465+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:23.465 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:23.465+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:23.718 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:23.466+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:23.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:23.720+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:23.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:23.720+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:23.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:23.720+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:23.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:23.721+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:24.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:24.176+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:24.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:24.176+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:24.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:24.176+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:24.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:24.177+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:24.773 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:24.694+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:24.774 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:24.694+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:24.774 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:24.694+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:24.774 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:24.695+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:24.775 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:24.486+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:24.775 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:24.486+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:24.775 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:24.486+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:24.775 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:24.487+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:25.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:10:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:10:25.263+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15901 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:10:25.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:25.178+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:25.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:25.178+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:25.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:25.178+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:25.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:25.179+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:25.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:25.660+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:25.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:25.660+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:25.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:25.660+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:25.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:25.660+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:25.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:25.480+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:25.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:25.480+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:25.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:25.480+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:25.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:25.481+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:26.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:26.153+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:26.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:26.153+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:26.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:26.153+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:26.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:26.153+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:26.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:26.625+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:26.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:26.625+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:26.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:26.625+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:26.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:26.626+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:26.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:26.478+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:26.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:26.478+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:26.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:26.478+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:26.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:26.479+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:27.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:27.167+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:27.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:27.167+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:27.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:27.167+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:27.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:27.167+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:27.722 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:27.656+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:27.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:27.656+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:27.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:27.656+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:27.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:27.656+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:27.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:27.471+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:27.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:27.471+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:27.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:27.471+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:27.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:27.472+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:28.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:28.216+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:28.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:28.216+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:28.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:28.216+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:28.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:28.217+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:28.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:28.701+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:28.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:28.701+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:28.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:28.701+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:28.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:28.702+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:28.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:28.501+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:28.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:28.501+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:28.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:28.501+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:28.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:28.502+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:29.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:29.193+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:29.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:29.193+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:29.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:29.193+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:29.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:29.194+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:29.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:29.525+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:29.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:29.525+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:29.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:29.525+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:29.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:29.525+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:29.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:29.717+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:29.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:29.717+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:29.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:29.717+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:29.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:29.718+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:30.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:10:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:10:30.265+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15982 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:10:30.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:30.160+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:30.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:30.160+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:30.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:30.160+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:30.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:30.161+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:30.917 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:30.708+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:30.917 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:30.708+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:30.917 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:30.709+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:30.917 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:30.709+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:30.917 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:30.512+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:30.917 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:30.512+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:30.918 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:30.512+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:30.918 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:30.512+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:31.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:31.131+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:31.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:31.131+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:31.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:31.131+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:31.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:31.132+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:31.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:31.553+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:31.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:31.553+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:31.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:31.553+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:31.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:31.553+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:31.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:31.696+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:31.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:31.696+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:31.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:31.696+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:31.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:31.697+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:32.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:32.160+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:32.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:32.160+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:32.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:32.160+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:32.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:32.161+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:32.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:32.675+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:32.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:32.675+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:32.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:32.675+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:32.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:32.676+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:32.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:32.549+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:32.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:32.549+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:32.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:32.549+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:32.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:32.549+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:33.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:33.203+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:33.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:33.203+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:33.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:33.203+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:33.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:33.204+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:33.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:33.631+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:33.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:33.631+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:33.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:33.631+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:33.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:33.632+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:33.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:33.523+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:33.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:33.523+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:33.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:33.523+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:33.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:33.523+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:34.560 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:34.255+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:34.560 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:34.255+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:34.561 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:34.255+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:34.561 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:34.256+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:34.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:34.561+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:34.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:34.561+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:34.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:34.561+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:34.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:34.561+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:34.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:34.601+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:34.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:34.601+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:34.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:34.601+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:34.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:34.602+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:35.562 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:10:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:10:35.266+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16066 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:10:35.562 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:35.276+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:35.562 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:35.276+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:35.563 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:35.276+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:35.563 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:35.277+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:35.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:35.589+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:35.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:35.589+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:35.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:35.589+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:35.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:35.590+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:35.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:35.563+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:35.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:35.563+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:35.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:35.563+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:35.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:35.564+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:36.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:36.292+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:36.560 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:36.292+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:36.560 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:36.292+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:36.560 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:36.293+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:36.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:36.561+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:36.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:36.561+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:36.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:36.561+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:36.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:36.561+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:36.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:36.578+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:36.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:36.578+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:36.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:36.578+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:36.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:36.579+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:37.585 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:37.315+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:37.585 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:37.315+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:37.586 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:37.315+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:37.586 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:37.316+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:37.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:37.608+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:37.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:37.608+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:37.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:37.608+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:37.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:37.609+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:37.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:37.586+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:37.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:37.586+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:37.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:37.586+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:37.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:37.587+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:38.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:38.309+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:38.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:38.309+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:38.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:38.309+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:38.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:38.310+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:38.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:38.559+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:38.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:38.559+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:38.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:38.559+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:38.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:38.560+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:38.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:38.572+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:38.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:38.572+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:38.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:38.572+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:38.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:38.573+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:39.518 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:39.288+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:39.519 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:39.288+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:39.519 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:39.288+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:39.519 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:39.289+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:39.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:39.519+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:39.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:39.519+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:39.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:39.519+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:39.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:39.520+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:39.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:39.622+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:39.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:39.622+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:39.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:39.622+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:39.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:39.623+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:40.391 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:10:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:10:40.267+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16148 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:10:40.392 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:40.302+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:40.392 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:40.302+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:40.392 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:40.302+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:40.392 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:40.303+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:40.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:40.558+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:40.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:40.558+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:40.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:40.558+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:40.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:40.559+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:40.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:40.633+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:40.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:40.633+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:40.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:40.633+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:40.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:40.634+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:41.574 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:41.288+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:41.575 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:41.288+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:41.575 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:41.288+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:41.575 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:41.289+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:41.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:41.652+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:41.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:41.652+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:41.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:41.652+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:41.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:41.653+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:41.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:41.574+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:41.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:41.574+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:41.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:41.574+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:41.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:41.575+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:42.585 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:42.242+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:42.585 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:42.242+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:42.586 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:42.242+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:42.586 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:42.242+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:42.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:42.628+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:42.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:42.628+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:42.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:42.628+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:42.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:42.628+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:42.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:42.585+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:42.977 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:42.585+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:42.977 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:42.585+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:42.977 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:42.586+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:43.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:43.195+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:43.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:43.195+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:43.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:43.195+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:43.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:43.195+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:43.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:43.607+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:43.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:43.607+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:43.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:43.607+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:43.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:43.608+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:43.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:43.607+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:43.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:43.607+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:43.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:43.607+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:43.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:43.608+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:44.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:44.189+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:44.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:44.189+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:44.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:44.189+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:44.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:44.189+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:44.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:44.632+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:44.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:44.632+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:44.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:44.632+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:44.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:44.633+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:44.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:44.613+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:44.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:44.613+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:44.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:44.613+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:44.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:44.613+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:45.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:10:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:10:45.269+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16233 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:10:45.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:45.208+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:45.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:45.208+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:45.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:45.208+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:45.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:45.209+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:45.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:45.621+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:45.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:45.621+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:45.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:45.621+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:45.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:45.622+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:45.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:45.643+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:45.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:45.643+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:45.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:45.643+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:45.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:45.643+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:46.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:46.190+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:46.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:46.190+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:46.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:46.190+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:46.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:46.191+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:46.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:46.667+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:46.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:46.667+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:46.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:46.667+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:46.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:46.668+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:46.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:46.675+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:46.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:46.675+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:46.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:46.675+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:46.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:46.675+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:47.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:47.179+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:47.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:47.179+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:47.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:47.179+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:47.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:47.179+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:47.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:47.643+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:47.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:47.643+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:47.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:47.643+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:47.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:47.643+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:47.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:47.677+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:47.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:47.677+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:47.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:47.677+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:47.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:47.678+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:48.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:48.187+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:48.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:48.187+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:48.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:48.187+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:48.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:48.188+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:48.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:48.641+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:48.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:48.641+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:48.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:48.641+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:48.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:48.641+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:48.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:48.653+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:48.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:48.653+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:48.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:48.653+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:48.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:48.654+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:49.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:49.139+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:49.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:49.139+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:49.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:49.139+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:49.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:49.140+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:49.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:49.665+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:49.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:49.665+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:49.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:49.665+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:49.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:49.666+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:49.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:49.635+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:49.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:49.635+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:49.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:49.635+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:49.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:49.636+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:50.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:10:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:10:50.271+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16315 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:10:50.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:50.177+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:50.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:50.177+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:50.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:50.177+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:50.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:50.178+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:50.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:50.700+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:50.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:50.700+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:50.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:50.700+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:50.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:50.700+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:50.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:50.616+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:50.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:50.616+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:50.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:50.616+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:50.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:50.617+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:51.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:51.215+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:51.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:51.215+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:51.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:51.215+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:51.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:51.216+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:51.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:51.717+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:51.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:51.717+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:51.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:51.717+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:51.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:51.719+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:51.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:51.611+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:51.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:51.611+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:51.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:51.611+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:51.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:51.612+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:52.606 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:52.227+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:52.606 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:52.227+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:52.606 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:52.227+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:52.606 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:52.227+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:52.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:52.606+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:52.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:52.606+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:52.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:52.606+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:52.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:52.607+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:52.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:52.680+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:52.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:52.680+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:52.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:52.680+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:52.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:52.681+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:53.572 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:53.233+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:53.572 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:53.233+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:53.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:53.233+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:53.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:53.234+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:53.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:53.632+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:53.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:53.632+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:53.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:53.632+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:53.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:53.633+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:53.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:53.673+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:53.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:53.673+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:53.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:53.673+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:53.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:53.673+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:54.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:54.203+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:54.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:54.203+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:54.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:54.203+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:54.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:54.204+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:54.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:54.666+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:54.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:54.666+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:54.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:54.666+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:54.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:54.666+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:54.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:54.699+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:54.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:54.699+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:54.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:54.699+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:54.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:54.699+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:55.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:10:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:10:55.273+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16401 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:10:55.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:55.172+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:55.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:55.172+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:55.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:55.172+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:55.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:55.172+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:55.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:55.659+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:55.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:55.659+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:55.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:55.659+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:55.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:55.660+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:55.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:55.650+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:55.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:55.650+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:55.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:55.650+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:55.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:55.651+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:56.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:56.132+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:56.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:56.132+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:56.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:56.132+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:56.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:56.132+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:56.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:56.652+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:56.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:56.652+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:56.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:56.652+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:56.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:56.652+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:56.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:56.709+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:56.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:56.709+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:56.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:56.709+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:56.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:56.709+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:57.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:57.106+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:57.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:57.106+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:57.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:57.106+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:57.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:57.106+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:57.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:57.724+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:57.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:57.724+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:57.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:57.724+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:57.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:57.725+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:57.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:57.631+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:57.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:57.631+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:57.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:57.631+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:57.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:57.632+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:58.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:58.128+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:58.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:58.128+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:58.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:58.128+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:58.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:58.129+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:58.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:58.770+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:58.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:58.770+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:58.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:58.770+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:58.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:58.771+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:58.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:58.673+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:58.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:58.673+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:58.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:58.673+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:58.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:58.674+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:59.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:59.157+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:59.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:59.157+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:59.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:59.157+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:10:59.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:10:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:10:59.158+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:10:59.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:59.777+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:59.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:59.777+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:59.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:59.777+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:10:59.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:10:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:10:59.778+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:10:59.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:59.691+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:59.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:59.691+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:59.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:59.691+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:10:59.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:10:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:10:59.692+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:00.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:11:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:11:00.275+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16484 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:11:00.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:00.191+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:00.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:00.191+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:00.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:00.191+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:00.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:00.192+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:01.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:00.800+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:01.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:00.800+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:01.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:00.800+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:01.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:00.800+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:01.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:00.727+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:01.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:00.727+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:01.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:00.727+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:01.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:00.728+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:01.637 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:01.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:01.637 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:01.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:01.637 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:01.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:01.637 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:01.241+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:01.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:01.687+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:01.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:01.687+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:01.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:01.687+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:01.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:01.688+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:01.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:01.805+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:01.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:01.806+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:01.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:01.806+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:01.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:01.807+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:02.661 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:02.260+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:02.661 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:02.260+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:02.662 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:02.260+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:02.662 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:02.261+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:03.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:02.771+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:03.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:02.771+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:03.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:02.771+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:03.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:02.772+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:03.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:02.727+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:03.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:02.727+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:03.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:02.727+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:03.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:02.727+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:03.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:03.271+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:03.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:03.271+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:03.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:03.271+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:03.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:03.272+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:03.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:03.748+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:03.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:03.748+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:03.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:03.748+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:03.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:03.749+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:03.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:03.722+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:03.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:03.722+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:03.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:03.722+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:03.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:03.723+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:04.705 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:04.283+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:04.706 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:04.283+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:04.706 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:04.283+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:04.706 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:04.283+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:04.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:04.766+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:04.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:04.766+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:04.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:04.766+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:04.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:04.767+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:04.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:04.707+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:04.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:04.707+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:04.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:04.707+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:04.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:04.707+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:05.724 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:11:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:11:05.277+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16570 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:11:05.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:05.252+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:05.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:05.252+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:05.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:05.252+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:05.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:05.253+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:06.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:05.815+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:06.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:05.815+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:06.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:05.815+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:06.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:05.815+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:06.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:05.740+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:06.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:05.740+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:06.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:05.740+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:06.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:05.740+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:06.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:06.250+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:06.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:06.250+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:06.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:06.250+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:06.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:06.251+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:07.222 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:06.737+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:07.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:06.737+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:07.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:06.737+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:07.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:06.738+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:07.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:06.841+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:07.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:06.841+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:07.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:06.841+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:07.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:06.842+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:07.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:07.223+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:07.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:07.223+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:07.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:07.223+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:07.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:07.224+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:08.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:07.826+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:08.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:07.826+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:08.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:07.826+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:08.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:07.827+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:08.203 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:07.726+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:08.203 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:07.726+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:08.203 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:07.726+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:08.204 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:07.727+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:08.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:08.204+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:08.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:08.204+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:08.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:08.204+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:08.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:08.205+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:09.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:08.784+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:09.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:08.784+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:09.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:08.784+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:09.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:08.784+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:09.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:08.764+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:09.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:08.764+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:09.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:08.764+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:09.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:08.765+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:09.722 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:09.230+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:09.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:09.230+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:09.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:09.230+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:09.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:09.230+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:10.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:09.766+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:10.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:09.766+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:10.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:09.766+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:10.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:09.767+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:10.161 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:09.789+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:10.161 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:09.789+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:10.161 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:09.789+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:10.161 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:09.789+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:10.473 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:11:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:11:10.279+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16653 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:11:10.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:10.191+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:10.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:10.191+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:10.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:10.191+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:10.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:10.192+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:11.206 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:10.781+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:11.206 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:10.781+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:11.206 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:10.781+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:11.206 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:10.782+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:11.207 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:10.729+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:11.207 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:10.729+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:11.207 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:10.729+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:11.207 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:10.730+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:11.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:11.207+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:11.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:11.207+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:11.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:11.207+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:11.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:11.208+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:11.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:11.707+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:11.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:11.707+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:11.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:11.707+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:11.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:11.707+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:11.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:11.783+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:11.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:11.783+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:11.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:11.783+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:11.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:11.783+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:12.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:12.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:12.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:12.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:12.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:12.198+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:12.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:12.199+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:12.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:12.740+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:12.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:12.740+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:12.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:12.740+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:12.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:12.741+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:12.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:12.701+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:12.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:12.701+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:12.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:12.701+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:12.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:12.702+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:13.574 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:13.233+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:13.574 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:13.233+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:13.574 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:13.233+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:13.575 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:13.233+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:13.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:13.701+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:13.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:13.701+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:13.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:13.701+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:13.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:13.702+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:13.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:13.734+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:13.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:13.734+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:13.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:13.734+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:13.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:13.734+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:14.703 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:14.272+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:14.703 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:14.272+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:14.703 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:14.272+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:14.703 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:14.273+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:14.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:14.704+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:14.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:14.704+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:14.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:14.704+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:14.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:14.705+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:14.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:14.710+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:14.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:14.710+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:14.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:14.710+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:14.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:14.711+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:15.694 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:11:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:11:15.280+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16739 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:11:15.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:15.322+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:15.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:15.322+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:15.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:15.322+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:15.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:15.323+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:15.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:15.695+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:15.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:15.695+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:15.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:15.695+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:15.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:15.696+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:15.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:15.726+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:15.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:15.726+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:15.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:15.726+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:15.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:15.727+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:16.693 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:16.329+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:16.694 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:16.329+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:16.694 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:16.329+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:16.694 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:16.330+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:16.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:16.735+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:16.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:16.735+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:16.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:16.735+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:16.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:16.736+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:16.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:16.695+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:16.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:16.695+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:16.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:16.695+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:16.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:16.695+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:17.690 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:17.290+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:17.690 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:17.290+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:17.691 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:17.290+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:17.691 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:17.292+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:17.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:17.691+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:17.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:17.691+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:17.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:17.691+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:17.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:17.692+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:17.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:17.703+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:17.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:17.703+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:17.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:17.703+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:17.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:17.704+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:18.665 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:18.288+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:18.665 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:18.288+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:18.665 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:18.288+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:18.666 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:18.289+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:18.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:18.713+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:18.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:18.713+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:18.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:18.713+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:18.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:18.714+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:18.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:18.666+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:18.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:18.666+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:18.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:18.666+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:18.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:18.667+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:19.688 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:19.245+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:19.688 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:19.245+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:19.688 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:19.245+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:19.689 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:19.246+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:19.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:19.711+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:19.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:19.711+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:19.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:19.711+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:19.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:19.711+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:19.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:19.689+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:19.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:19.689+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:19.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:19.689+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:19.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:19.690+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:20.405 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:11:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:11:20.282+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16822 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:11:20.405 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:20.266+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:20.406 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:20.266+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:20.406 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:20.266+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:20.406 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:20.267+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:20.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:20.670+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:20.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:20.670+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:20.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:20.670+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:20.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:20.671+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:20.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:20.707+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:20.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:20.707+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:20.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:20.707+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:20.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:20.707+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:21.665 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:21.251+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:21.666 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:21.251+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:21.666 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:21.251+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:21.666 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:21.252+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:21.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:21.667+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:21.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:21.667+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:21.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:21.667+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:21.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:21.668+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:21.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:21.753+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:21.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:21.753+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:21.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:21.753+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:21.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:21.753+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:22.682 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:22.278+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:22.682 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:22.278+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:22.682 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:22.278+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:22.682 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:22.279+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:22.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:22.683+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:22.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:22.683+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:22.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:22.683+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:22.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:22.684+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:22.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:22.782+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:22.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:22.782+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:22.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:22.782+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:22.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:22.783+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:23.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:23.272+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:23.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:23.272+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:23.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:23.272+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:23.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:23.273+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:23.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:23.659+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:23.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:23.659+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:23.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:23.659+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:23.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:23.660+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:23.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:23.801+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:23.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:23.801+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:23.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:23.801+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:23.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:23.802+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:24.692 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:24.300+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:24.692 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:24.300+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:24.692 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:24.300+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:24.692 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:24.301+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:24.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:24.693+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:24.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:24.693+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:24.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:24.693+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:24.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:24.694+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:24.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:24.772+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:24.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:24.772+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:24.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:24.772+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:24.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:24.773+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:25.702 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:11:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:11:25.283+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16908 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:11:25.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:25.335+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:25.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:25.335+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:25.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:25.335+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:25.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:25.335+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:25.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:25.758+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:25.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:25.758+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:25.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:25.758+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:25.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:25.759+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:25.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:25.703+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:25.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:25.703+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:25.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:25.703+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:25.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:25.704+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:26.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:26.307+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:26.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:26.307+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:26.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:26.307+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:26.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:26.308+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:26.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:26.664+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:26.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:26.664+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:26.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:26.664+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:26.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:26.665+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:26.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:26.753+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:26.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:26.753+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:26.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:26.753+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:26.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:26.754+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:27.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:27.327+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:27.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:27.327+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:27.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:27.327+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:27.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:27.327+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:27.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:27.725+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:27.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:27.725+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:27.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:27.725+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:27.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:27.725+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:27.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:27.702+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:27.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:27.702+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:27.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:27.702+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:27.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:27.703+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:28.693 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:28.344+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:28.694 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:28.344+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:28.694 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:28.344+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:28.694 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:28.344+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:28.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:28.695+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:28.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:28.695+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:28.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:28.695+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:28.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:28.696+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:28.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:28.741+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:28.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:28.741+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:28.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:28.741+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:28.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:28.742+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:29.669 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:29.331+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:29.669 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:29.331+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:29.669 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:29.331+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:29.669 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:29.331+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:29.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:29.714+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:29.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:29.714+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:29.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:29.714+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:29.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:29.714+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:29.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:29.670+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:29.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:29.670+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:29.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:29.670+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:29.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:29.671+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:30.209 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-23T09:11:30.210 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2024-01-23T09:11:30.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T09:11:30.211 DEBUG:teuthology.orchestra.run.smithi059:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2024-01-23T09:11:30.243 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/pidns/pidns02.c: Cannot open: Input/output error 2024-01-23T09:11:30.244 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/pidns/pidns06.c: Cannot open: Input/output error 2024-01-23T09:11:30.244 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/pidns/pidns12.c: Cannot open: Input/output error 2024-01-23T09:11:30.244 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/pidns/pidns10.c: Cannot open: Input/output error 2024-01-23T09:11:30.244 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/pidns/pidns01.c: Cannot open: Input/output error 2024-01-23T09:11:30.244 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/pidns/Makefile: Cannot open: Input/output error 2024-01-23T09:11:30.245 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/pidns/runpidnstest.sh: Cannot open: Input/output error 2024-01-23T09:11:30.245 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/pidns/pidns17.c: Cannot open: Input/output error 2024-01-23T09:11:30.245 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/pidns/pidns13.c: Cannot open: Input/output error 2024-01-23T09:11:30.245 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/pidns/pidns31.c: Cannot open: Input/output error 2024-01-23T09:11:30.245 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/pidns/README: Cannot open: Input/output error 2024-01-23T09:11:30.245 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/pidns/pidns03.c: Cannot open: Input/output error 2024-01-23T09:11:30.245 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/pidns/pidns04.c: Cannot open: Input/output error 2024-01-23T09:11:30.247 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/pidns/pidns20.c: Cannot open: Input/output error 2024-01-23T09:11:30.247 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/pidns/pidns16.c: Cannot open: Input/output error 2024-01-23T09:11:30.247 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/pidns/pidns05.c: Cannot open: Input/output error 2024-01-23T09:11:30.247 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/pidns/pidns30.c: Cannot open: Input/output error 2024-01-23T09:11:30.247 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/pidns: Cannot utime: Input/output error 2024-01-23T09:11:30.248 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/libclone: Cannot mkdir: Input/output error 2024-01-23T09:11:30.248 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/libclone/Makefile: Cannot open: Input/output error 2024-01-23T09:11:30.248 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/libclone/libclone.c: Cannot open: Input/output error 2024-01-23T09:11:30.248 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/libclone/libclone.h: Cannot open: Input/output error 2024-01-23T09:11:30.248 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns: Cannot mkdir: Input/output error 2024-01-23T09:11:30.248 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/delchild.sh: Cannot open: Input/output error 2024-01-23T09:11:30.248 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/parent_view.sh: Cannot open: Input/output error 2024-01-23T09:11:30.248 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/parentns.sh: Cannot open: Input/output error 2024-01-23T09:11:30.249 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/two_children_ns.c: Cannot open: Input/output error 2024-01-23T09:11:30.249 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/paripv6.sh: Cannot open: Input/output error 2024-01-23T09:11:30.249 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/common.c: Cannot open: Input/output error 2024-01-23T09:11:30.249 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/par_chld_ipv6.c: Cannot open: Input/output error 2024-01-23T09:11:30.249 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/readme: Cannot open: Input/output error 2024-01-23T09:11:30.249 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/check_netns_enabled.c: Cannot open: Input/output error 2024-01-23T09:11:30.249 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/parent_2.sh: Cannot open: Input/output error 2024-01-23T09:11:30.249 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/parent.sh: Cannot open: Input/output error 2024-01-23T09:11:30.249 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/parent_share.sh: Cannot open: Input/output error 2024-01-23T09:11:30.250 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/parent_1.sh: Cannot open: Input/output error 2024-01-23T09:11:30.250 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/Makefile: Cannot open: Input/output error 2024-01-23T09:11:30.250 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/runnetnstest.sh: Cannot open: Input/output error 2024-01-23T09:11:30.250 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/child_2.sh: Cannot open: Input/output error 2024-01-23T09:11:30.250 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/child_1.sh: Cannot open: Input/output error 2024-01-23T09:11:30.250 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/child.sh: Cannot open: Input/output error 2024-01-23T09:11:30.250 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/childns.sh: Cannot open: Input/output error 2024-01-23T09:11:30.250 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/crtchild_delchild.c: Cannot open: Input/output error 2024-01-23T09:11:30.251 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/runallnetworktests_parent.sh: Cannot open: Input/output error 2024-01-23T09:11:30.251 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/initialize.sh: Cannot open: Input/output error 2024-01-23T09:11:30.251 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/nw_under_ns.sh: Cannot open: Input/output error 2024-01-23T09:11:30.251 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/par_ftp.sh: Cannot open: Input/output error 2024-01-23T09:11:30.251 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/sysfsview.c: Cannot open: Input/output error 2024-01-23T09:11:30.251 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/par_chld_ftp.c: Cannot open: Input/output error 2024-01-23T09:11:30.251 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/runallnetworktests_child.sh: Cannot open: Input/output error 2024-01-23T09:11:30.251 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/rename_net.sh: Cannot open: Input/output error 2024-01-23T09:11:30.252 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/create_container.c: Cannot open: Input/output error 2024-01-23T09:11:30.252 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/childipv6.sh: Cannot open: Input/output error 2024-01-23T09:11:30.252 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/crtchild.c: Cannot open: Input/output error 2024-01-23T09:11:30.252 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/ch_ftp.sh: Cannot open: Input/output error 2024-01-23T09:11:30.252 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/container_ftp.pl: Cannot open: Input/output error 2024-01-23T09:11:30.252 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/netns/child_propagate.sh: Cannot open: Input/output error 2024-01-23T09:11:30.252 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers/Makefile.inc: Cannot open: Input/output error 2024-01-23T09:11:30.252 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/containers: Cannot utime: Input/output error 2024-01-23T09:11:30.253 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io: Cannot mkdir: Input/output error 2024-01-23T09:11:30.253 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/stress_cd: Cannot mkdir: Input/output error 2024-01-23T09:11:30.253 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/stress_cd/stress_cd.c: Cannot open: Input/output error 2024-01-23T09:11:30.253 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/stress_cd/Makefile: Cannot open: Input/output error 2024-01-23T09:11:30.253 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/stress_floppy: Cannot mkdir: Input/output error 2024-01-23T09:11:30.256 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/stress_floppy/dd_file: Cannot open: Input/output error 2024-01-23T09:11:30.256 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/stress_floppy/stress_floppy: Cannot open: Input/output error 2024-01-23T09:11:30.256 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/stress_floppy/Makefile: Cannot open: Input/output error 2024-01-23T09:11:30.256 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/stress_floppy/generate.sh: Cannot open: Input/output error 2024-01-23T09:11:30.257 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/stress_floppy/1K_file: Cannot open: Input/output error 2024-01-23T09:11:30.257 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest: Cannot mkdir: Input/output error 2024-01-23T09:11:30.257 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/parse.h: Cannot open: Input/output error 2024-01-23T09:11:30.257 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/man1: Cannot mkdir: Input/output error 2024-01-23T09:11:30.257 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/man1/disktest.1.gz: Cannot open: Input/output error 2024-01-23T09:11:30.257 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/man1/disktest_manual.html: Cannot open: Input/output error 2024-01-23T09:11:30.257 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/man1/disktest.1: Cannot open: Input/output error 2024-01-23T09:11:30.258 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/Makefile.aix: Cannot open: Input/output error 2024-01-23T09:11:30.258 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/disktest.spec: Cannot open: Input/output error 2024-01-23T09:11:30.258 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/usage.h: Cannot open: Input/output error 2024-01-23T09:11:30.258 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/Getopt.c: Cannot open: Input/output error 2024-01-23T09:11:30.258 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/CHANGELOG: Cannot open: Input/output error 2024-01-23T09:11:30.258 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/io.c: Cannot open: Input/output error 2024-01-23T09:11:30.258 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/timer.c: Cannot open: Input/output error 2024-01-23T09:11:30.258 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/Makefile.linux: Cannot open: Input/output error 2024-01-23T09:11:30.259 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/LICENSE: Cannot open: Input/output error 2024-01-23T09:11:30.259 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/threading.h: Cannot open: Input/output error 2024-01-23T09:11:30.259 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/childmain.h: Cannot open: Input/output error 2024-01-23T09:11:30.259 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/dump.h: Cannot open: Input/output error 2024-01-23T09:11:30.259 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/Makefile: Cannot open: Input/output error 2024-01-23T09:11:30.259 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/sfunc.c: Cannot open: Input/output error 2024-01-23T09:11:30.260 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-01-23T09:11:30.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-23T09:11:30.261 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-23T09:11:30.277 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-23T09:11:30.277 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/parse.c: Cannot open: Input/output error 2024-01-23T09:11:30.277 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/signals.h: Cannot open: Input/output error 2024-01-23T09:11:30.277 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/childmain.c: Cannot open: Input/output error 2024-01-23T09:11:30.278 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/sfunc.h: Cannot open: Input/output error 2024-01-23T09:11:30.278 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/dump.c: Cannot open: Input/output error 2024-01-23T09:11:30.278 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/README: Cannot open: Input/output error 2024-01-23T09:11:30.278 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/timer.h: Cannot open: Input/output error 2024-01-23T09:11:30.278 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/Makefile.windows: Cannot open: Input/output error 2024-01-23T09:11:30.278 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/io.h: Cannot open: Input/output error 2024-01-23T09:11:30.278 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/stats.h: Cannot open: Input/output error 2024-01-23T09:11:30.278 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/usage.c: Cannot open: Input/output error 2024-01-23T09:11:30.279 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/threading.c: Cannot open: Input/output error 2024-01-23T09:11:30.279 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/main.h: Cannot open: Input/output error 2024-01-23T09:11:30.279 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/globals.h: Cannot open: Input/output error 2024-01-23T09:11:30.279 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/signals.c: Cannot open: Input/output error 2024-01-23T09:11:30.279 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/main.c: Cannot open: Input/output error 2024-01-23T09:11:30.279 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/defs.h: Cannot open: Input/output error 2024-01-23T09:11:30.279 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/stats.c: Cannot open: Input/output error 2024-01-23T09:11:30.279 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/globals.c: Cannot open: Input/output error 2024-01-23T09:11:30.280 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/disktest/Getopt.h: Cannot open: Input/output error 2024-01-23T09:11:30.280 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/aio: Cannot mkdir: Input/output error 2024-01-23T09:11:30.280 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/aio/Makefile: Cannot open: Input/output error 2024-01-23T09:11:30.280 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/aio/aio02: Cannot mkdir: Input/output error 2024-01-23T09:11:30.280 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/aio/aio02/cases: Cannot mkdir: Input/output error 2024-01-23T09:11:30.280 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/aio/aio02/cases/common.h: Cannot open: Input/output error 2024-01-23T09:11:30.280 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/aio/aio02/Makefile: Cannot open: Input/output error 2024-01-23T09:11:30.280 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/aio/aio02/runfstests.sh: Cannot open: Input/output error 2024-01-23T09:11:30.281 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/aio/aio02/README: Cannot open: Input/output error 2024-01-23T09:11:30.281 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/aio/aio02/aio_tio.c: Cannot open: Input/output error 2024-01-23T09:11:30.281 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/aio/aio02/common.h: Cannot open: Input/output error 2024-01-23T09:11:30.281 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/aio/aio02/main.c: Cannot open: Input/output error 2024-01-23T09:11:30.281 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/aio/aio01: Cannot mkdir: Input/output error 2024-01-23T09:11:30.281 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/aio/aio01/Makefile: Cannot open: Input/output error 2024-01-23T09:11:30.281 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/aio/aio01/aio01.c: Cannot open: Input/output error 2024-01-23T09:11:30.281 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/writetest: Cannot mkdir: Input/output error 2024-01-23T09:11:30.282 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/writetest/Makefile: Cannot open: Input/output error 2024-01-23T09:11:30.282 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/writetest/writetest.c: Cannot open: Input/output error 2024-01-23T09:11:30.282 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/writetest/gpl.txt: Cannot open: Input/output error 2024-01-23T09:11:30.282 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/Makefile: Cannot open: Input/output error 2024-01-23T09:11:30.282 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/direct_io: Cannot mkdir: Input/output error 2024-01-23T09:11:30.282 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/direct_io/diotest_routines.h: Cannot open: Input/output error 2024-01-23T09:11:30.282 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/direct_io/test_dma_thread_diotest7.sh: Cannot open: Input/output error 2024-01-23T09:11:30.282 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/direct_io/diotest6.c: Cannot open: Input/output error 2024-01-23T09:11:30.282 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/direct_io/diotest3.c: Cannot open: Input/output error 2024-01-23T09:11:30.283 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/direct_io/Makefile: Cannot open: Input/output error 2024-01-23T09:11:30.283 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/direct_io/diotest4.c: Cannot open: Input/output error 2024-01-23T09:11:30.283 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/direct_io/diotest1.c: Cannot open: Input/output error 2024-01-23T09:11:30.283 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/direct_io/README: Cannot open: Input/output error 2024-01-23T09:11:30.283 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/direct_io/diotest_routines.c: Cannot open: Input/output error 2024-01-23T09:11:30.283 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/direct_io/diotest2.c: Cannot open: Input/output error 2024-01-23T09:11:30.283 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/direct_io/dma_thread_diotest7.c: Cannot open: Input/output error 2024-01-23T09:11:30.283 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/direct_io/diotest5.c: Cannot open: Input/output error 2024-01-23T09:11:30.283 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/ltp-aiodio: Cannot mkdir: Input/output error 2024-01-23T09:11:30.283 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/ltp-aiodio/aio-stress.c: Cannot open: Input/output error 2024-01-23T09:11:30.284 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/ltp-aiodio/yesenter.txt: Cannot open: Input/output error 2024-01-23T09:11:30.284 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/ltp-aiodio/Makefile: Cannot open: Input/output error 2024-01-23T09:11:30.284 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/ltp-aiodio/dirty.c: Cannot open: Input/output error 2024-01-23T09:11:30.284 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/ltp-aiodio/read_checkzero.c: Cannot open: Input/output error 2024-01-23T09:11:30.284 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/ltp-aiodio/README: Cannot open: Input/output error 2024-01-23T09:11:30.284 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/ltp-aiodio/dio_sparse.c: Cannot open: Input/output error 2024-01-23T09:11:30.284 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/ltp-aiodio/aiodio_sparse.c: Cannot open: Input/output error 2024-01-23T09:11:30.284 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/ltp-aiodio/ltp-diorh.c: Cannot open: Input/output error 2024-01-23T09:11:30.285 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/ltp-aiodio/dio_append.c: Cannot open: Input/output error 2024-01-23T09:11:30.285 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/ltp-aiodio/aiodio_append.c: Cannot open: Input/output error 2024-01-23T09:11:30.285 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/ltp-aiodio/aiocp.c: Cannot open: Input/output error 2024-01-23T09:11:30.285 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/io/ltp-aiodio/dio_truncate.c: Cannot open: Input/output error 2024-01-23T09:11:30.285 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/Makefile: Cannot open: Input/output error 2024-01-23T09:11:30.285 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers: Cannot mkdir: Input/output error 2024-01-23T09:11:30.285 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/nls: Cannot mkdir: Input/output error 2024-01-23T09:11:30.285 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/nls/Makefile: Cannot open: Input/output error 2024-01-23T09:11:30.286 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/nls/userBlockNLS.c: Cannot open: Input/output error 2024-01-23T09:11:30.286 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/nls/README: Cannot open: Input/output error 2024-01-23T09:11:30.286 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/nls/nlsTest.c: Cannot open: Input/output error 2024-01-23T09:11:30.286 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/nls/nlsTest.h: Cannot open: Input/output error 2024-01-23T09:11:30.286 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci: Cannot mkdir: Input/output error 2024-01-23T09:11:30.286 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci/tpci: Cannot mkdir: Input/output error 2024-01-23T09:11:30.286 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci/tpci/Makefile: Cannot open: Input/output error 2024-01-23T09:11:30.286 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci/tpci/tpci.mod.c: Cannot open: Input/output error 2024-01-23T09:11:30.286 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci/tpci/tpci.h: Cannot open: Input/output error 2024-01-23T09:11:30.287 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci/tpci/tpci.c: Cannot open: Input/output error 2024-01-23T09:11:30.287 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci/tpci/st_tpci.h: Cannot open: Input/output error 2024-01-23T09:11:30.287 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci/tpci/pci-exp-tests-HOWTO: Cannot open: Input/output error 2024-01-23T09:11:30.287 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci/user_tpci: Cannot mkdir: Input/output error 2024-01-23T09:11:30.287 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci/user_tpci/tpci_ki.c: Cannot open: Input/output error 2024-01-23T09:11:30.287 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci/user_tpci/user_tpci.c: Cannot open: Input/output error 2024-01-23T09:11:30.287 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci/user_tpci/Makefile: Cannot open: Input/output error 2024-01-23T09:11:30.287 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/pci/user_tpci/user_tpci.h: Cannot open: Input/output error 2024-01-23T09:11:30.288 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/usb: Cannot mkdir: Input/output error 2024-01-23T09:11:30.288 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/usb/tusb: Cannot mkdir: Input/output error 2024-01-23T09:11:30.288 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/usb/tusb/st_tusb.h: Cannot open: Input/output error 2024-01-23T09:11:30.288 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/usb/tusb/tusb.h: Cannot open: Input/output error 2024-01-23T09:11:30.288 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/usb/tusb/Makefile: Cannot open: Input/output error 2024-01-23T09:11:30.288 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/usb/tusb/README: Cannot open: Input/output error 2024-01-23T09:11:30.288 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/usb/tusb/tusb.c: Cannot open: Input/output error 2024-01-23T09:11:30.289 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/usb/user_usb: Cannot mkdir: Input/output error 2024-01-23T09:11:30.289 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/usb/user_usb/user_tusb.c: Cannot open: Input/output error 2024-01-23T09:11:30.289 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/usb/user_usb/Makefile: Cannot open: Input/output error 2024-01-23T09:11:30.289 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/usb/user_usb/tusb_ki.c: Cannot open: Input/output error 2024-01-23T09:11:30.289 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/usb/user_usb/user_tusb.h: Cannot open: Input/output error 2024-01-23T09:11:30.289 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base: Cannot mkdir: Input/output error 2024-01-23T09:11:30.289 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base/user_base: Cannot mkdir: Input/output error 2024-01-23T09:11:30.289 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base/user_base/Makefile: Cannot open: Input/output error 2024-01-23T09:11:30.290 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base/user_base/user_tbase.c: Cannot open: Input/output error 2024-01-23T09:11:30.290 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base/user_base/user_tbase.h: Cannot open: Input/output error 2024-01-23T09:11:30.290 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base/user_base/tbase_ki.c: Cannot open: Input/output error 2024-01-23T09:11:30.290 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base/tbase: Cannot mkdir: Input/output error 2024-01-23T09:11:30.290 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base/tbase/tbase.mod.c: Cannot open: Input/output error 2024-01-23T09:11:30.290 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base/tbase/tbase.c: Cannot open: Input/output error 2024-01-23T09:11:30.290 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base/tbase/Makefile: Cannot open: Input/output error 2024-01-23T09:11:30.290 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base/tbase/tbase.h: Cannot open: Input/output error 2024-01-23T09:11:30.291 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base/tbase/load_tbase.sh: Cannot open: Input/output error 2024-01-23T09:11:30.291 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/base/tbase/str_mod.h: Cannot open: Input/output error 2024-01-23T09:11:30.291 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework: Cannot mkdir: Input/output error 2024-01-23T09:11:30.291 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/user_space: Cannot mkdir: Input/output error 2024-01-23T09:11:30.291 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/user_space/Makefile: Cannot open: Input/output error 2024-01-23T09:11:30.291 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/user_space/user_tmod.c: Cannot open: Input/output error 2024-01-23T09:11:30.291 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/user_space/tmod_ki.c: Cannot open: Input/output error 2024-01-23T09:11:30.292 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/user_space/user_tmod.h: Cannot open: Input/output error 2024-01-23T09:11:30.292 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/kernel_space: Cannot mkdir: Input/output error 2024-01-23T09:11:30.292 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/kernel_space/load_tmod.sh: Cannot open: Input/output error 2024-01-23T09:11:30.292 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/kernel_space/Makefile: Cannot open: Input/output error 2024-01-23T09:11:30.292 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/kernel_space/tmod.h: Cannot open: Input/output error 2024-01-23T09:11:30.292 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/kernel_space/str_mod.h: Cannot open: Input/output error 2024-01-23T09:11:30.292 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/kernel_space/tmod.c: Cannot open: Input/output error 2024-01-23T09:11:30.292 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/kernel_space/tmod.mod.c: Cannot open: Input/output error 2024-01-23T09:11:30.293 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/dev_sim_framework/README: Cannot open: Input/output error 2024-01-23T09:11:30.293 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/README: Cannot open: Input/output error 2024-01-23T09:11:30.293 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l: Cannot mkdir: Input/output error 2024-01-23T09:11:30.293 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space: Cannot mkdir: Input/output error 2024-01-23T09:11:30.293 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_QUERYCTRL.c: Cannot open: Input/output error 2024-01-23T09:11:30.293 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_EXT_CTRLS.h: Cannot open: Input/output error 2024-01-23T09:11:30.293 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_ENUM_FRAMESIZES.c: Cannot open: Input/output error 2024-01-23T09:11:30.293 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_PARM.h: Cannot open: Input/output error 2024-01-23T09:11:30.294 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/v4l2_test.c: Cannot open: Input/output error 2024-01-23T09:11:30.294 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_QUERYMENU.h: Cannot open: Input/output error 2024-01-23T09:11:30.294 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_ENUMINPUT.c: Cannot open: Input/output error 2024-01-23T09:11:30.294 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_QUERYBUF.c: Cannot open: Input/output error 2024-01-23T09:11:30.294 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_INPUT.h: Cannot open: Input/output error 2024-01-23T09:11:30.294 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/dev_video.h: Cannot open: Input/output error 2024-01-23T09:11:30.294 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_FREQUENCY.h: Cannot open: Input/output error 2024-01-23T09:11:30.294 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_OUTPUT.h: Cannot open: Input/output error 2024-01-23T09:11:30.294 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_FREQUENCY.c: Cannot open: Input/output error 2024-01-23T09:11:30.295 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_AUDOUT.h: Cannot open: Input/output error 2024-01-23T09:11:30.295 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_ENUMAUDOUT.h: Cannot open: Input/output error 2024-01-23T09:11:30.295 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_ENUMAUDIO.h: Cannot open: Input/output error 2024-01-23T09:11:30.295 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/test_VIDIOC_CTRL.h: Cannot open: Input/output error 2024-01-23T09:11:30.295 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/doc: Cannot mkdir: Input/output error 2024-01-23T09:11:30.295 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/doc/results.html: Cannot open: Input/output error 2024-01-23T09:11:30.295 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/doc/index.html: Cannot open: Input/output error 2024-01-23T09:11:30.295 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/device-drivers/v4l/user_space/doc/spec: Cannot mkdir 2024-01-23T09:11:30.295 INFO:tasks.workunit.client.0.smithi059.stderr:gzip: : Input/output error 2024-01-23T09:11:30.295 INFO:tasks.workunit.client.0.smithi059.stderr:stdin: Stale file handle 2024-01-23T09:11:30.296 INFO:tasks.workunit.client.0.smithi059.stderr:tar: Unexpected EOF in archive 2024-01-23T09:11:30.296 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel: Cannot utime: Input/output error 2024-01-23T09:11:30.296 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases: Cannot utime: Input/output error 2024-01-23T09:11:30.296 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231: Cannot utime: Input/output error 2024-01-23T09:11:30.296 INFO:tasks.workunit.client.0.smithi059.stderr:tar: Error is not recoverable: exiting now 2024-01-23T09:11:30.297 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-01-23T09:11:30.297 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-01-23T09:11:30.364 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-23T09:11:30.364 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-23T09:11:30.414 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:11:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:11:30.285+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16991 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:11:30.414 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:30.296+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:30.414 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:30.296+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:30.414 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:30.296+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:30.414 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:30.297+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:30.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:30.677+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:30.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:30.677+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:30.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:30.677+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:30.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:30.678+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:30.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:30.684+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:30.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:30.685+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:30.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:30.685+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:30.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:30.685+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:31.633 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:31.271+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:31.633 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:31.271+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:31.633 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:31.271+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:31.633 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:31.273+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:31.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:31.707+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:31.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:31.707+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:31.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:31.708+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:31.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:31.708+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:31.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:31.634+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:31.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:31.634+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:31.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:31.634+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:31.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:31.634+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:32.643 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:32.278+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:32.643 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:32.278+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:32.644 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:32.278+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:32.644 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:32.279+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:32.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:32.708+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:32.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:32.708+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:32.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:32.708+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:32.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:32.708+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:32.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:32.644+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:32.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:32.644+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:32.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:32.644+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:32.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:32.645+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:33.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:33.309+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:33.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:33.309+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:33.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:33.309+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:33.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:33.310+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:33.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:33.709+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:33.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:33.709+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:33.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:33.709+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:33.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:33.710+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:33.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:33.663+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:33.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:33.663+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:33.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:33.663+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:33.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:33.664+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:34.623 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:34.270+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:34.623 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:34.270+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:34.624 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:34.270+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:34.624 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:34.271+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:34.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:34.624+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:34.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:34.624+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:34.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:34.624+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:34.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:34.625+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:34.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:34.704+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:34.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:34.704+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:34.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:34.704+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:34.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:34.705+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:35.630 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:11:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:11:35.287+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17077 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:11:35.630 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:35.250+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:35.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:35.250+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:35.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:35.250+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:35.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:35.251+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:35.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:35.631+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:35.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:35.631+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:35.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:35.631+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:35.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:35.632+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:35.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:35.755+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:35.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:35.755+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:35.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:35.755+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:35.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:35.756+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:36.630 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:36.290+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:36.630 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:36.290+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:36.630 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:36.290+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:36.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:36.290+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:36.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:36.631+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:36.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:36.631+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:36.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:36.631+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:36.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:36.631+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:36.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:36.709+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:36.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:36.709+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:36.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:36.709+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:36.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:36.709+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:37.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:37.264+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:37.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:37.264+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:37.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:37.264+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:37.592 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:37.264+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:37.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:37.592+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:37.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:37.592+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:37.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:37.592+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:37.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:37.593+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:37.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:37.716+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:37.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:37.716+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:37.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:37.716+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:37.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:37.717+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:38.595 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:38.253+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:38.596 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:38.253+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:38.596 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:38.253+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:38.596 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:38.253+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:38.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:38.597+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:38.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:38.597+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:38.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:38.597+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:38.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:38.597+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:38.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:38.722+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:38.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:38.722+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:38.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:38.722+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:38.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:38.723+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:39.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:39.222+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:39.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:39.222+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:39.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:39.222+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:39.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:39.223+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:39.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:39.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:39.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:39.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:39.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:39.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:39.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:39.627+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:39.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:39.706+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:39.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:39.706+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:39.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:39.706+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:39.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:39.707+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:40.413 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:11:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:11:40.289+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17160 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:11:40.413 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:40.265+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:40.413 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:40.265+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:40.414 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:40.265+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:40.414 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:40.267+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:40.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:40.609+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:40.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:40.609+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:40.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:40.609+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:40.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:40.610+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:41.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:40.726+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:41.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:40.726+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:41.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:40.726+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:41.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:40.727+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:41.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:41.260+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:41.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:41.260+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:41.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:41.260+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:41.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:41.261+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:41.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:41.612+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:41.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:41.612+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:41.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:41.612+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:41.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:41.613+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:41.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:41.751+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:41.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:41.751+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:41.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:41.751+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:41.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:41.751+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:42.644 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:42.253+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:42.645 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:42.253+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:42.645 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:42.253+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:42.645 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:42.253+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:42.972 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:42.645+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:42.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:42.645+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:42.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:42.645+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:42.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:42.646+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:42.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:42.787+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:42.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:42.787+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:42.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:42.787+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:42.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:42.787+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:43.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:43.242+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:43.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:43.242+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:43.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:43.242+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:43.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:43.243+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:43.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:43.794+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:43.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:43.794+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:43.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:43.794+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:43.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:43.795+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:43.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:43.597+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:43.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:43.597+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:43.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:43.597+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:43.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:43.598+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:44.575 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:44.239+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:44.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:44.239+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:44.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:44.239+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:44.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:44.240+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:44.830 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:44.576+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:44.831 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:44.576+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:44.831 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:44.576+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:44.831 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:44.577+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:45.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:44.831+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:45.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:44.831+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:45.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:44.831+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:45.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:44.832+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:45.619 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:11:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:11:45.290+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17246 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:11:45.619 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:45.276+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:45.620 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:45.276+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:45.620 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:45.276+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:45.620 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:45.277+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:45.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:45.620+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:45.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:45.620+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:45.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:45.620+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:45.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:45.620+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:45.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:45.795+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:45.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:45.795+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:45.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:45.795+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:45.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:45.795+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:46.586 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:46.243+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:46.586 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:46.243+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:46.586 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:46.243+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:46.586 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:46.244+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:46.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:46.826+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:46.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:46.826+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:46.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:46.826+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:46.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:46.827+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:46.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:46.587+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:46.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:46.587+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:46.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:46.587+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:46.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:46.588+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:47.636 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:47.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:47.636 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:47.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:47.636 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:47.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:47.636 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:47.241+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:47.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:47.839+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:47.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:47.839+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:47.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:47.839+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:47.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:47.840+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:47.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:47.637+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:47.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:47.637+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:47.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:47.637+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:47.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:47.637+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:48.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:48.203+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:48.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:48.203+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:48.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:48.203+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:48.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:48.204+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:48.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:48.815+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:48.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:48.815+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:48.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:48.815+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:48.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:48.815+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:48.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:48.649+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:48.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:48.649+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:48.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:48.649+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:48.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:48.649+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:49.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:49.193+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:49.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:49.193+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:49.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:49.193+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:49.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:49.194+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:49.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:49.637+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:49.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:49.637+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:49.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:49.637+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:49.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:49.637+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:49.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:49.800+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:49.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:49.800+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:49.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:49.800+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:49.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:49.800+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:50.414 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:11:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:11:50.292+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17329 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:11:50.415 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:50.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:50.415 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:50.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:50.415 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:50.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:50.415 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:50.242+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:50.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:50.603+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:50.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:50.603+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:50.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:50.603+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:50.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:50.604+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:51.210 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:50.844+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:51.210 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:50.844+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:51.210 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:50.844+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:51.211 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:50.845+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:51.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:51.211+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:51.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:51.211+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:51.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:51.211+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:51.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:51.211+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:51.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:51.830+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:51.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:51.830+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:51.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:51.830+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:51.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:51.831+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:51.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:51.605+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:51.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:51.605+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:51.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:51.605+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:51.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:51.606+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:52.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:52.195+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:52.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:52.195+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:52.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:52.195+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:52.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:52.195+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:52.910 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:52.785+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:52.910 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:52.785+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:52.910 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:52.785+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:52.910 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:52.786+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:52.911 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:52.618+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:52.911 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:52.618+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:52.911 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:52.618+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:52.911 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:52.618+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:53.577 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:53.234+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:53.577 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:53.234+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:53.577 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:53.234+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:53.577 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:53.234+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:53.861 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:53.810+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:53.861 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:53.810+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:53.861 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:53.810+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:53.862 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:53.810+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:53.862 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:53.578+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:53.862 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:53.578+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:53.862 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:53.578+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:53.862 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:53.579+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:54.564 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:54.232+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:54.564 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:54.232+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:54.564 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:54.232+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:54.565 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:54.233+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:54.817 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:54.565+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:54.818 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:54.565+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:54.818 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:54.565+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:54.818 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:54.566+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:55.187 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:54.818+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:55.188 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:54.818+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:55.188 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:54.818+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:55.188 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:54.819+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:55.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:11:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:11:55.293+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17414 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:11:55.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:55.188+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:55.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:55.188+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:55.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:55.188+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:55.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:55.189+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:55.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:55.856+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:55.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:55.856+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:55.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:55.856+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:55.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:55.857+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:55.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:55.612+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:55.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:55.612+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:55.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:55.612+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:55.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:55.613+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:56.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:56.202+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:56.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:56.202+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:56.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:56.202+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:56.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:56.203+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:56.862 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:56.597+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:56.863 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:56.597+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:56.863 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:56.597+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:56.863 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:56.598+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:57.185 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:56.864+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:57.186 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:56.864+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:57.186 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:56.864+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:57.186 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:56.865+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:57.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:57.187+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:57.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:57.187+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:57.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:57.187+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:57.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:57.188+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:57.873 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:57.603+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:57.873 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:57.603+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:57.873 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:57.603+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:57.874 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:57.604+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:58.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:57.874+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:58.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:57.874+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:58.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:57.874+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:58.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:57.875+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:58.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:58.201+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:58.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:58.201+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:58.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:58.201+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:58.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:58.202+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:58.917 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:58.633+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:58.918 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:58.633+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:58.918 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:58.633+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:58.918 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:58.633+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:59.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:58.917+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:59.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:58.917+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:59.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:58.917+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:11:59.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:58.918+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:11:59.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:59.200+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:59.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:59.200+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:59.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:59.200+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:11:59.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:11:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:11:59.201+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:11:59.928 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:59.589+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:59.929 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:59.589+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:59.929 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:59.589+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:11:59.929 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:11:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:11:59.590+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:00.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:59.928+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:00.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:59.928+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:00.204 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:59.928+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:00.204 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:11:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:11:59.929+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:00.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:12:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:12:00.294+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17496 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:12:00.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:00.203+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:00.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:00.203+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:00.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:00.203+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:00.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:00.204+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:00.885 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:00.567+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:00.885 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:00.567+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:00.885 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:00.567+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:00.885 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:00.567+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:01.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:00.885+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:01.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:00.885+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:01.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:00.885+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:01.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:00.886+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:01.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:01.236+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:01.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:01.236+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:01.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:01.236+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:01.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:01.237+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:01.935 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:01.576+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:01.935 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:01.576+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:01.935 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:01.576+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:01.936 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:01.577+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:02.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:01.935+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:02.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:01.935+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:02.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:01.935+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:02.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:01.936+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:02.557 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:02.247+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:02.557 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:02.247+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:02.557 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:02.247+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:02.557 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:02.248+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:02.918 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:02.557+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:02.918 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:02.557+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:02.918 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:02.557+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:02.918 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:02.557+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:03.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:02.918+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:03.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:02.918+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:03.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:02.918+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:03.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:02.919+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:03.577 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:03.234+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:03.577 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:03.234+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:03.577 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:03.234+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:03.577 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:03.235+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:03.896 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:03.577+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:03.897 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:03.577+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:03.897 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:03.577+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:03.897 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:03.578+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:04.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:03.896+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:04.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:03.896+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:04.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:03.896+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:04.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:03.897+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:04.579 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:04.250+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:04.580 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:04.250+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:04.580 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:04.250+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:04.580 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:04.251+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:04.848 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:04.579+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:04.848 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:04.579+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:04.849 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:04.579+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:04.849 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:04.580+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:05.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:04.848+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:05.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:04.848+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:05.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:04.848+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:05.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:04.849+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:05.530 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:12:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:12:05.296+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17581 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:12:05.530 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:05.295+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:05.531 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:05.295+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:05.531 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:05.295+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:05.531 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:05.297+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:05.845 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:05.531+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:05.845 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:05.531+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:05.846 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:05.531+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:05.846 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:05.532+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:06.096 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:05.845+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:06.096 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:05.845+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:06.096 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:05.845+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:06.096 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:05.846+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:06.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:06.303+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:06.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:06.303+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:06.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:06.303+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:06.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:06.304+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:06.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:06.533+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:06.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:06.533+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:06.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:06.533+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:06.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:06.533+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:07.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:06.811+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:07.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:06.811+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:07.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:06.811+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:07.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:06.812+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:07.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:07.517+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:07.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:07.517+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:07.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:07.517+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:07.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:07.518+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:07.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:07.270+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:07.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:07.270+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:07.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:07.270+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:07.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:07.271+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:08.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:07.773+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:08.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:07.773+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:08.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:07.773+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:08.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:07.774+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:08.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:08.471+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:08.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:08.471+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:08.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:08.471+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:08.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:08.472+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:08.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:08.245+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:08.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:08.245+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:08.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:08.245+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:08.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:08.246+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:09.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:08.783+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:09.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:08.783+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:09.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:08.783+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:09.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:08.784+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:09.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:09.251+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:09.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:09.251+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:09.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:09.251+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:09.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:09.252+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:09.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:09.429+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:09.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:09.429+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:09.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:09.429+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:09.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:09.430+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:10.165 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:09.748+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:10.165 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:09.748+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:10.166 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:09.748+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:10.166 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:09.748+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:10.422 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:12:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:12:10.298+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17663 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:12:10.423 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:10.222+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:10.423 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:10.222+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:10.423 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:10.222+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:10.423 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:10.223+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:10.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:10.436+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:10.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:10.436+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:10.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:10.436+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:10.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:10.436+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:11.183 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:10.776+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:11.183 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:10.776+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:11.183 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:10.776+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:11.184 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:10.776+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:11.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:11.421+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:11.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:11.421+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:11.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:11.421+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:11.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:11.421+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:11.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:11.183+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:11.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:11.183+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:11.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:11.183+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:11.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:11.184+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:11.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:11.775+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:11.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:11.775+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:11.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:11.775+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:11.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:11.776+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:12.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:12.425+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:12.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:12.425+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:12.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:12.425+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:12.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:12.426+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:12.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:12.186+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:12.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:12.186+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:12.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:12.186+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:12.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:12.187+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:13.147 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:12.798+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:13.148 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:12.798+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:13.148 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:12.798+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:13.148 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:12.799+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:13.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:13.147+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:13.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:13.147+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:13.456 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:13.147+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:13.456 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:13.148+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:13.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:13.456+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:13.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:13.456+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:13.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:13.456+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:13.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:13.456+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:14.151 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:13.825+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:14.151 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:13.825+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:14.151 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:13.825+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:14.151 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:13.826+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:14.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:14.151+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:14.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:14.151+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:14.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:14.151+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:14.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:14.152+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:14.835 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:14.483+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:14.835 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:14.483+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:14.835 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:14.483+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:14.835 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:14.483+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:15.112 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:14.835+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:15.112 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:14.835+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:15.112 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:14.835+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:15.112 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:14.836+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:15.365 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:12:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:12:15.300+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17748 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:12:15.365 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:15.113+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:15.365 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:15.113+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:15.366 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:15.113+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:15.366 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:15.113+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:15.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:15.465+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:15.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:15.465+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:15.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:15.465+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:15.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:15.466+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:16.151 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:15.814+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:16.151 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:15.814+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:16.151 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:15.814+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:16.152 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:15.815+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:16.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:16.151+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:16.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:16.151+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:16.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:16.151+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:16.456 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:16.152+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:16.730 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:16.491+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:16.731 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:16.491+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:16.731 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:16.491+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:16.731 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:16.492+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:17.111 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:16.775+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:17.111 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:16.775+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:17.112 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:16.775+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:17.112 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:16.776+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:17.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:17.154+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:17.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:17.154+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:17.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:17.154+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:17.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:17.154+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:17.777 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:17.495+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:17.777 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:17.495+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:17.777 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:17.495+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:17.778 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:17.496+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:18.189 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:17.777+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:18.189 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:17.777+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:18.189 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:17.777+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:18.190 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:17.778+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:18.448 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:18.190+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:18.448 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:18.190+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:18.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:18.190+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:18.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:18.191+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:18.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:18.449+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:18.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:18.449+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:18.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:18.449+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:18.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:18.450+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:19.171 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:18.758+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:19.172 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:18.758+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:19.172 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:18.758+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:19.172 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:18.758+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:19.424 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:19.172+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:19.424 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:19.172+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:19.424 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:19.172+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:19.424 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:19.174+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:19.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:19.425+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:19.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:19.425+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:19.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:19.425+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:19.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:19.426+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:20.165 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:19.784+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:20.165 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:19.784+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:20.166 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:19.784+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:20.166 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:19.785+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:20.427 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:12:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:12:20.303+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17830 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:12:20.428 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:20.183+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:20.428 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:20.183+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:20.428 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:20.183+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:20.428 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:20.184+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:20.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:20.442+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:20.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:20.442+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:20.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:20.442+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:20.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:20.443+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:21.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:20.817+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:21.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:20.817+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:21.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:20.817+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:21.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:20.818+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:21.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:21.160+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:21.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:21.160+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:21.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:21.160+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:21.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:21.161+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:21.803 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:21.487+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:21.803 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:21.487+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:21.803 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:21.487+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:21.804 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:21.488+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:22.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:21.804+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:22.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:21.804+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:22.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:21.804+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:22.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:21.805+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:22.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:22.201+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:22.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:22.201+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:22.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:22.201+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:22.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:22.202+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:22.794 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:22.526+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:22.794 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:22.526+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:22.794 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:22.526+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:22.794 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:22.527+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:23.184 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:22.795+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:23.184 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:22.795+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:23.184 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:22.795+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:23.184 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:22.796+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:23.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:23.185+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:23.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:23.185+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:23.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:23.185+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:23.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:23.186+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:23.807 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:23.490+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:23.807 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:23.490+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:23.807 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:23.490+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:23.808 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:23.490+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:24.158 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:23.808+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:24.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:23.808+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:24.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:23.808+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:24.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:23.809+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:24.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:24.159+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:24.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:24.159+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:24.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:24.159+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:24.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:24.160+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:24.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:24.448+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:24.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:24.448+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:24.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:24.448+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:24.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:24.448+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:25.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:24.788+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:25.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:24.788+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:25.204 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:24.788+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:25.204 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:24.789+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:25.472 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:12:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:12:25.304+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17915 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:12:25.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:25.202+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:25.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:25.202+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:25.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:25.202+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:25.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:25.203+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:25.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:25.422+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:25.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:25.422+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:25.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:25.422+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:25.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:25.422+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:26.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:25.775+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:26.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:25.775+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:26.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:25.775+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:26.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:25.776+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:26.454 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:26.203+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:26.454 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:26.203+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:26.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:26.203+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:26.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:26.204+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:26.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:26.455+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:26.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:26.455+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:26.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:26.455+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:26.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:26.456+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:27.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:26.824+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:27.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:26.824+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:27.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:26.824+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:27.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:26.825+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:27.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:27.421+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:27.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:27.421+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:27.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:27.421+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:27.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:27.421+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:27.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:27.237+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:27.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:27.237+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:27.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:27.237+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:27.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:27.238+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:28.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:27.778+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:28.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:27.778+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:28.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:27.778+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:28.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:27.778+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:28.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:28.247+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:28.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:28.247+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:28.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:28.247+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:28.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:28.248+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:28.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:28.396+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:28.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:28.396+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:28.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:28.396+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:28.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:28.397+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:29.218 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:28.805+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:29.219 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:28.805+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:29.219 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:28.805+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:29.219 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:28.806+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:29.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:29.219+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:29.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:29.220+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:29.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:29.220+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:29.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:29.221+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:29.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:29.399+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:29.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:29.399+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:29.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:29.399+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:29.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:29.399+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:30.166 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:29.832+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:30.166 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:29.832+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:30.166 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:29.832+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:30.166 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:29.833+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:30.421 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:12:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:12:30.306+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17997 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:12:30.422 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:30.229+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:30.422 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:30.229+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:30.422 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:30.229+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:30.422 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:30.230+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:30.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:30.422+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:30.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:30.422+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:30.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:30.422+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:30.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:30.423+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:31.182 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:30.790+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:31.183 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:30.790+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:31.183 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:30.790+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:31.183 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:30.790+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:31.433 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:31.184+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:31.433 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:31.184+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:31.433 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:31.184+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:31.433 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:31.185+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:31.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:31.434+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:31.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:31.434+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:31.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:31.434+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:31.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:31.434+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:31.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:31.819+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:31.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:31.819+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:31.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:31.819+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:31.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:31.820+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:32.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:32.388+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:32.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:32.388+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:32.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:32.388+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:32.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:32.388+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:32.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:32.158+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:32.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:32.158+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:32.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:32.158+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:32.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:32.159+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:33.124 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:32.794+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:33.125 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:32.794+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:33.125 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:32.794+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:33.125 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:32.795+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:33.375 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:33.126+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:33.375 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:33.126+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:33.375 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:33.126+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:33.375 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:33.126+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:33.376 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:33.375+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:33.376 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:33.375+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:33.376 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:33.375+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:33.376 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:33.376+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:34.084 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:33.810+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:34.085 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:33.810+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:34.085 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:33.810+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:34.085 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:33.810+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:34.401 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:34.085+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:34.401 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:34.085+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:34.401 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:34.085+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:34.401 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:34.087+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:34.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:34.402+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:34.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:34.402+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:34.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:34.402+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:34.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:34.402+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:35.096 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:34.827+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:35.096 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:34.827+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:35.097 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:34.827+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:35.097 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:34.828+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:35.430 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:12:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:12:35.308+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18079 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:12:35.431 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:35.097+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:35.431 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:35.097+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:35.431 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:35.097+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:35.431 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:35.098+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:35.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:35.441+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:35.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:35.441+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:35.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:35.441+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:35.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:35.441+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:36.071 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:35.800+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:36.071 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:35.800+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:36.072 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:35.800+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:36.072 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:35.801+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:36.394 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:36.072+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:36.394 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:36.072+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:36.394 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:36.072+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:36.394 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:36.073+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:36.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:36.395+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:36.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:36.395+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:36.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:36.395+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:36.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:36.396+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:37.087 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:36.825+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:37.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:36.825+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:37.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:36.825+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:37.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:36.826+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:37.403 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:37.088+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:37.403 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:37.088+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:37.403 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:37.088+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:37.403 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:37.089+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:37.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:37.404+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:37.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:37.404+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:37.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:37.404+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:37.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:37.404+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:38.075 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:37.812+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:38.075 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:37.812+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:38.075 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:37.812+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:38.075 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:37.812+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:38.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:38.076+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:38.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:38.076+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:38.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:38.076+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:38.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:38.077+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:38.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:38.419+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:38.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:38.419+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:38.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:38.419+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:38.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:38.420+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:39.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:39.107+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:39.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:39.107+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:39.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:39.107+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:39.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:39.108+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:39.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:38.859+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:39.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:38.859+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:39.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:38.859+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:39.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:38.860+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:39.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:39.436+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:39.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:39.436+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:39.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:39.436+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:39.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:39.437+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:40.115 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:39.854+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:40.115 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:39.854+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:40.115 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:39.854+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:40.115 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:39.855+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:40.423 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:12:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:12:40.310+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18162 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:12:40.423 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:40.116+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:40.424 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:40.116+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:40.424 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:40.116+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:40.424 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:40.117+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:40.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:40.424+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:40.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:40.424+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:40.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:40.424+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:40.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:40.425+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:41.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:40.886+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:41.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:40.886+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:41.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:40.886+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:41.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:40.887+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:41.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:41.123+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:41.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:41.123+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:41.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:41.123+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:41.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:41.124+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:41.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:41.383+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:41.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:41.383+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:41.703 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:41.383+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:41.703 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:41.384+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:41.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:41.855+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:41.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:41.855+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:41.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:41.855+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:41.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:41.857+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:42.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:42.157+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:42.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:42.157+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:42.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:42.157+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:42.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:42.158+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:42.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:42.366+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:42.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:42.366+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:42.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:42.366+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:42.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:42.366+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:43.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:42.869+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:43.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:42.869+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:43.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:42.869+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:43.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:42.869+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:43.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:43.320+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:43.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:43.320+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:43.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:43.320+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:43.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:43.321+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:43.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:43.193+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:43.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:43.193+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:43.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:43.193+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:43.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:43.195+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:44.189 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:43.848+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:44.189 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:43.848+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:44.189 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:43.848+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:44.190 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:43.849+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:44.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:44.309+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:44.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:44.309+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:44.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:44.309+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:44.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:44.310+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:44.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:44.190+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:44.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:44.190+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:44.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:44.190+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:44.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:44.191+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:45.173 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:44.865+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:45.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:44.865+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:45.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:44.865+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:45.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:44.867+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:45.433 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:12:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:12:45.311+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18246 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:12:45.433 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:45.175+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:45.433 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:45.175+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:45.433 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:45.175+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:45.434 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:45.175+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:45.434 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:45.282+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:45.434 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:45.282+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:45.434 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:45.282+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:45.434 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:45.283+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:46.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:45.869+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:46.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:45.869+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:46.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:45.869+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:46.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:45.869+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:46.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:46.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:46.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:46.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:46.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:46.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:46.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:46.198+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:46.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:46.289+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:46.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:46.289+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:46.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:46.289+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:46.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:46.290+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:47.219 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:46.890+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:47.219 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:46.890+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:47.219 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:46.890+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:47.220 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:46.890+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:47.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:47.220+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:47.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:47.220+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:47.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:47.220+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:47.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:47.221+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:47.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:47.310+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:47.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:47.310+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:47.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:47.310+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:47.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:47.310+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:48.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:47.853+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:48.252 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:47.853+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:48.252 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:47.853+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:48.252 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:47.854+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:48.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:48.266+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:48.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:48.266+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:48.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:48.266+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:48.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:48.267+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:48.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:48.335+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:48.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:48.335+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:48.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:48.335+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:48.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:48.335+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:49.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:48.877+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:49.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:48.877+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:49.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:48.877+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:49.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:48.877+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:49.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:49.223+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:49.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:49.223+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:49.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:49.223+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:49.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:49.224+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:49.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:49.335+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:49.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:49.335+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:49.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:49.335+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:49.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:49.335+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:50.168 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:49.888+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:50.168 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:49.888+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:50.168 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:49.888+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:50.168 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:49.889+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:50.436 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:12:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:12:50.313+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18328 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:12:50.436 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:50.263+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:50.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:50.263+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:50.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:50.263+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:50.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:50.263+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:50.437 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:50.370+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:50.437 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:50.370+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:50.437 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:50.370+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:50.438 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:50.371+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:51.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:50.873+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:51.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:50.873+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:51.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:50.873+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:51.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:50.874+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:51.703 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:51.266+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:51.703 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:51.266+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:51.703 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:51.266+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:51.703 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:51.267+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:51.704 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:51.332+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:51.704 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:51.332+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:51.704 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:51.332+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:51.704 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:51.332+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:51.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:51.827+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:51.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:51.827+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:51.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:51.827+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:51.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:51.828+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:52.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:52.351+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:52.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:52.351+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:52.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:52.351+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:52.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:52.351+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:52.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:52.315+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:52.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:52.315+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:52.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:52.315+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:52.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:52.316+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:53.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:52.872+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:53.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:52.872+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:53.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:52.872+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:53.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:52.873+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:53.580 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:53.327+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:53.580 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:53.327+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:53.580 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:53.327+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:53.581 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:53.328+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:53.581 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:53.353+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:53.581 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:53.353+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:53.581 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:53.353+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:53.581 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:53.353+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:54.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:53.871+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:54.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:53.871+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:54.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:53.871+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:54.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:53.872+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:54.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:54.357+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:54.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:54.357+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:54.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:54.357+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:54.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:54.358+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:54.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:54.345+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:54.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:54.345+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:54.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:54.345+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:22.263120+0000 (oldest deadline 2024-01-23T09:02:42.263120+0000) 2024-01-23T09:12:54.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:54.345+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:55.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:54.880+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:55.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:54.880+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:55.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:54.880+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:55.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:54.881+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:55.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:12:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:12:55.315+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18411 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:12:55.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:55.357+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:55.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:55.331+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:55.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:55.331+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:55.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:55.331+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:55.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:55.332+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:56.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:55.912+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:56.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:55.912+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:56.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:55.912+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:56.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:55.913+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:56.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:56.322+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:56.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:56.315+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:56.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:56.315+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:56.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:56.315+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:56.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:56.316+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:57.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:56.900+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:57.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:56.900+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:57.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:56.900+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:57.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:56.901+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:57.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:57.305+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:57.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:57.304+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:57.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:57.304+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:57.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:57.304+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:57.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:57.304+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:58.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:57.896+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:58.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:57.896+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:58.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:57.896+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:58.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:57.897+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:58.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:58.265+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:58.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:58.308+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:58.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:58.308+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:58.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:58.308+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:58.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:58.308+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:59.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:58.861+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:59.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:58.861+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:59.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:58.861+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:12:59.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:58.862+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:12:59.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:12:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:12:59.262+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:12:59.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:59.272+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:59.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:59.272+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:59.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:59.272+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:12:59.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:12:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:12:59.272+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:00.169 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:59.826+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:00.169 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:59.826+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:00.169 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:59.826+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:00.169 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:12:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:12:59.827+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:00.436 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:13:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:13:00.316+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18493 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:13:00.437 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:00.302+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:00.439 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:00.279+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:13:00.439 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:00.279+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:13:00.439 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:00.279+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:13:00.439 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:00.280+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:01.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:00.822+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:01.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:00.822+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:01.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:00.822+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:01.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:00.823+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:01.704 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:01.287+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:01.704 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:01.238+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:13:01.704 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:01.238+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:13:01.704 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:01.238+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:34.144196+0000 (oldest deadline 2024-01-23T09:02:54.144196+0000) 2024-01-23T09:13:01.705 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:01.238+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:01.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:01.839+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:01.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:01.839+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:01.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:01.839+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:01.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:01.840+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:02.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:02.285+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:02.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:02.288+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:03.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:02.859+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:03.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:02.859+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:03.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:02.859+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:03.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:02.860+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:03.581 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:03.262+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:03.581 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:03.263+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:03.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:03.829+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:03.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:03.829+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:03.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:03.829+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:03.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:03.830+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:04.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:04.220+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:04.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:04.233+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:05.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:04.784+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:05.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:04.784+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:05.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:04.784+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:05.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:04.785+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:05.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:05.237+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:05.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:13:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:13:05.318+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18576 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:13:05.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:05.266+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:06.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:05.822+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:06.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:05.822+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:06.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:05.822+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:06.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:05.823+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:06.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:06.264+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:06.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:06.230+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:07.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:06.849+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:07.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:06.849+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:07.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:06.849+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:07.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:06.850+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:07.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:07.247+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:07.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:07.199+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:08.149 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:07.846+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:08.149 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:07.846+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:08.150 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:07.846+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:08.150 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:07.847+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:08.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:08.267+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:08.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:08.150+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:09.123 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:08.807+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:09.123 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:08.807+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:09.123 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:08.807+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:09.123 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:08.808+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:09.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:09.301+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:09.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:09.124+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:10.102 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:09.826+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:10.102 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:09.826+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:10.102 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:09.826+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:10.103 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:09.827+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:10.444 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:13:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:13:10.320+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18658 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:13:10.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:10.278+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:10.444 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:10.103+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:11.139 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:10.807+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:11.139 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:10.807+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:11.140 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:10.807+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:11.140 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:10.808+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:11.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:11.140+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:11.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:11.269+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:11.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:11.848+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:11.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:11.848+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:11.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:11.848+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:11.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:11.849+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:12.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:12.232+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:12.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:12.099+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:13.148 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:12.895+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:13.148 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:12.895+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:13.148 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:12.895+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:02:37.817547+0000 (oldest deadline 2024-01-23T09:02:57.817547+0000) 2024-01-23T09:13:13.148 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:12.896+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:13.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:13.230+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:13.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:13.149+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:14.188 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:13.913+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:14.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:14.256+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:14.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:14.189+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:15.177 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:14.887+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:15.444 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:13:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:13:15.322+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18741 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:13:15.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:15.219+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:15.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:15.178+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:15.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:15.178+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:15.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:15.178+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:15.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:15.179+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:16.137 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:15.887+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:16.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:16.228+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:16.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:16.138+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:16.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:16.138+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:16.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:16.138+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:16.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:16.139+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:17.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:16.919+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:17.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:17.102+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:17.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:17.102+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:17.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:17.102+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:17.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:17.102+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:17.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:17.225+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:18.208 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:17.929+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:18.208 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:18.076+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:18.208 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:18.076+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:18.208 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:18.076+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:18.209 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:18.077+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:18.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:18.209+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:19.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:18.971+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:19.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:19.170+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:19.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:19.028+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:19.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:19.028+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:19.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:19.028+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:19.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:19.029+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:20.207 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:19.951+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:20.207 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:20.025+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:20.208 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:20.025+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:20.208 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:20.025+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:20.208 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:20.025+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:20.473 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:13:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:13:20.323+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18824 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:13:20.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:20.207+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:21.215 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:20.921+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:21.215 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:21.045+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:21.216 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:21.045+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:21.216 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:21.045+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:21.216 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:21.046+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:21.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:21.215+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:22.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:21.960+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:22.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:22.065+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:22.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:22.065+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:22.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:22.065+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:22.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:22.065+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:22.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:22.256+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:23.210 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:22.987+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:23.211 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:23.115+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:23.211 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:23.115+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:23.211 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:23.115+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:23.211 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:23.115+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:23.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:23.210+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:23.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:23.210+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:23.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:23.210+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:23.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:23.211+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:24.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:23.979+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:24.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:24.186+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:24.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:24.186+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:24.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:24.186+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:24.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:24.187+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:24.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:24.161+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:24.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:24.162+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:24.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:24.162+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:24.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:24.162+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:25.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:24.951+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:25.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:25.176+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:25.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:25.176+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:25.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:25.176+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:25.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:25.177+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:25.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:25.172+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:25.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:25.172+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:25.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:25.172+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:25.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:25.172+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:25.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:13:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:13:25.325+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18907 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:13:26.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:25.986+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:26.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:26.189+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:26.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:26.189+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:26.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:26.189+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:26.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:26.190+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:26.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:26.145+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:26.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:26.145+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:26.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:26.145+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:26.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:26.146+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:27.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:27.016+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:27.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:27.115+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:27.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:27.115+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:27.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:27.115+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:27.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:27.115+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:27.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:27.235+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:27.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:27.235+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:27.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:27.235+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:27.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:27.236+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:28.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:28.067+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:28.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:28.079+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:28.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:28.079+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:28.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:28.079+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:28.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:28.080+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:28.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:28.249+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:28.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:28.249+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:28.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:28.249+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:28.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:28.250+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:29.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:29.042+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:29.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:29.239+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:29.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:29.239+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:29.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:29.239+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:29.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:29.240+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:29.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:29.073+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:29.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:29.073+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:29.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:29.073+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:29.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:29.074+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:30.327 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:30.091+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:30.327 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:30.272+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:30.328 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:30.272+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:30.328 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:30.272+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:30.328 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:30.273+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:30.328 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:30.032+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:30.328 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:30.032+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:30.329 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:30.032+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:30.329 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:30.033+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:30.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:13:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:13:30.327+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18990 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:13:31.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:31.101+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:31.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:31.224+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:31.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:31.224+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:31.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:31.224+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:31.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:31.224+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:31.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:31.056+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:31.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:31.056+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:31.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:31.056+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:31.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:31.057+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:32.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:32.062+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:32.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:32.262+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:32.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:32.262+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:32.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:32.262+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:32.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:32.263+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:32.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:32.082+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:32.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:32.082+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:32.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:32.082+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:32.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:32.082+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:33.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:33.077+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:33.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:33.109+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:33.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:33.109+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:33.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:33.109+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:33.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:33.110+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:33.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:33.310+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:33.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:33.310+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:33.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:33.310+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:33.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:33.310+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:34.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:34.111+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:34.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:34.111+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:34.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:34.111+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:34.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:34.112+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:34.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:34.339+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:34.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:34.339+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:34.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:34.339+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:34.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:34.339+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:34.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:34.133+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:34.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:34.133+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:34.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:34.133+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:34.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:34.134+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:35.329 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:35.074+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:35.329 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:35.074+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:35.329 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:35.074+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:35.330 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:35.075+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:35.330 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:35.316+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:35.330 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:35.316+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:35.330 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:35.316+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:35.330 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:35.316+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:35.331 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:35.086+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:35.331 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:35.086+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:35.331 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:35.086+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:35.331 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:35.087+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:35.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:13:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:13:35.328+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19074 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:13:36.292 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:36.033+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:36.292 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:36.033+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:36.292 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:36.033+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:36.292 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:36.033+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:36.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:36.091+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:36.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:36.091+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:36.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:36.091+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:36.294 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:36.092+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:36.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:36.293+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:36.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:36.293+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:36.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:36.293+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:36.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:36.293+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:37.303 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:37.020+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:37.303 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:37.020+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:37.303 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:37.020+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:37.303 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:37.020+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:37.304 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:37.078+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:37.304 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:37.078+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:37.304 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:37.078+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:37.304 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:37.079+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:37.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:37.303+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:37.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:37.303+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:37.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:37.303+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:37.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:37.304+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:38.333 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:38.069+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:38.334 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:38.069+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:38.334 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:38.069+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:38.334 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:38.070+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:38.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:38.128+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:38.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:38.128+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:38.334 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:38.128+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:38.335 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:38.129+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:38.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:38.333+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:38.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:38.333+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:38.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:38.333+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:38.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:38.334+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:39.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:39.170+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:39.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:39.170+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:39.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:39.170+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:39.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:39.171+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:39.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:39.112+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:39.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:39.112+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:39.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:39.112+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:39.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:39.113+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:39.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:39.322+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:39.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:39.322+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:39.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:39.322+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:39.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:39.323+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:40.331 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:40.067+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:40.331 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:40.067+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:40.331 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:40.067+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:40.331 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:40.068+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:40.332 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:40.197+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:40.332 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:40.197+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:40.332 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:40.197+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:40.332 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:40.198+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:40.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:13:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:13:40.331+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19156 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:13:40.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:40.364+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:40.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:40.364+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:40.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:40.364+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:40.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:40.365+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:41.346 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:41.043+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:41.346 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:41.043+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:41.347 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:41.043+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:41.347 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:41.044+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:41.347 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:41.221+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:41.348 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:41.221+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:41.348 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:41.221+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:41.348 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:41.222+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:41.708 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:41.347+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:41.708 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:41.347+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:41.708 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:41.347+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:41.708 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:41.347+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:42.330 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:42.243+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:42.331 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:42.243+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:42.331 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:42.243+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:42.331 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:42.244+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:42.331 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:42.012+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:42.331 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:42.012+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:42.331 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:42.012+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:42.332 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:42.013+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:42.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:42.331+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:42.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:42.331+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:42.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:42.331+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:42.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:42.332+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:43.308 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:43.057+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:43.308 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:43.057+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:43.309 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:43.057+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:43.309 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:43.058+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:43.309 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:43.207+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:43.309 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:43.207+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:43.309 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:43.207+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:43.309 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:43.207+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:43.585 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:43.309+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:43.585 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:43.309+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:43.585 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:43.309+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:43.585 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:43.310+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:44.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:44.058+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:44.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:44.058+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:44.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:44.058+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:44.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:44.059+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:44.210+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:44.210+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:44.210+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:44.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:44.211+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:44.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:44.338+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:44.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:44.338+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:44.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:44.338+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:44.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:44.339+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:45.325 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:45.040+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:45.325 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:45.040+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:45.326 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:45.040+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:45.326 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:45.041+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:45.326 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:45.170+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:45.326 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:45.170+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:45.326 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:45.170+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:45.326 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:45.171+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:45.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:13:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:13:45.333+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19241 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:13:45.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:45.326+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:45.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:45.326+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:45.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:45.326+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:45.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:45.327+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:46.376 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:46.055+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:46.376 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:46.055+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:46.377 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:46.055+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:46.377 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:46.057+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:46.377 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:46.126+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:46.377 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:46.126+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:46.379 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:46.126+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:46.379 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:46.127+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:46.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:46.377+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:46.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:46.377+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:46.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:46.377+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:46.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:46.377+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:47.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:47.118+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:47.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:47.118+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:47.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:47.118+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:47.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:47.119+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:47.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:47.338+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:47.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:47.338+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:47.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:47.338+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:47.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:47.339+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:47.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:47.095+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:47.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:47.095+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:47.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:47.095+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:47.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:47.096+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:48.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:48.117+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:48.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:48.117+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:48.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:48.117+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:48.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:48.117+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:48.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:48.302+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:48.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:48.302+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:48.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:48.302+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:48.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:48.303+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:48.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:48.072+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:48.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:48.072+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:48.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:48.072+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:48.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:48.072+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:49.328 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:49.070+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:49.328 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:49.070+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:49.328 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:49.070+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:49.328 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:49.071+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:49.328 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:49.091+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:49.329 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:49.091+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:49.329 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:49.091+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:49.329 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:49.092+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:49.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:49.328+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:49.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:49.328+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:49.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:49.328+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:49.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:49.330+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:50.335 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:50.066+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:50.335 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:50.066+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:50.335 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:50.066+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:50.335 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:50.067+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:50.336 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:50.288+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:50.336 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:50.288+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:50.336 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:50.288+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:50.336 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:50.289+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:50.336 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:50.043+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:50.336 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:50.043+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:50.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:50.043+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:50.337 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:50.043+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:50.597 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:13:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:13:50.335+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19325 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:13:51.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:51.289+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:51.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:51.289+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:51.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:51.289+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:51.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:51.290+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:51.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:51.091+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:51.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:51.091+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:51.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:51.091+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:51.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:51.092+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:51.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:51.083+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:51.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:51.083+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:51.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:51.083+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:51.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:51.084+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:52.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:52.273+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:52.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:52.273+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:52.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:52.273+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:52.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:52.273+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:52.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:52.047+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:52.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:52.047+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:52.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:52.047+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:52.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:52.047+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:52.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:52.117+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:52.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:52.117+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:52.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:52.117+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:52.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:52.117+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:53.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:53.258+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:53.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:53.258+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:53.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:53.258+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:53.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:53.259+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:53.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:53.035+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:53.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:53.035+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:53.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:53.035+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:53.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:53.036+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:53.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:53.115+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:53.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:53.115+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:53.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:53.115+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:53.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:53.115+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:54.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:54.029+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:54.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:54.029+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:54.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:54.029+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:54.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:54.030+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:54.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:54.246+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:54.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:54.246+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:54.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:54.246+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:54.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:54.247+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:54.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:54.069+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:54.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:54.069+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:54.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:54.069+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:54.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:54.069+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:55.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:55.063+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:55.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:55.063+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:55.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:55.063+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:55.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:55.064+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:55.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:55.242+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:55.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:55.242+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:55.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:55.242+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:55.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:55.243+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:55.338 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:55.055+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:55.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:55.055+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:55.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:55.055+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:55.339 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:55.056+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:55.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:13:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:13:55.337+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19410 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:13:56.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:56.231+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:56.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:56.231+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:56.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:56.231+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:56.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:56.232+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:56.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:56.022+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:56.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:56.022+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:56.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:56.022+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:56.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:56.023+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:56.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:56.063+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:56.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:56.063+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:56.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:56.063+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:56.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:56.064+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:57.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:57.211+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:57.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:57.211+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:57.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:57.211+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:57.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:57.211+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:57.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:57.054+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:57.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:57.054+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:57.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:57.054+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:57.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:57.054+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:57.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:57.099+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:57.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:57.099+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:57.477 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:57.099+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:57.477 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:57.100+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:58.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:58.198+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:58.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:58.198+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:58.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:58.198+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:58.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:58.199+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:58.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:58.127+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:58.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:58.127+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:58.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:58.127+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:58.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:58.128+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:58.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:58.058+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:58.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:58.058+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:58.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:58.058+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:58.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:58.059+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:13:59.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:59.152+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:59.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:59.152+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:59.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:59.152+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:13:59.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:13:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:13:59.152+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:59.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:59.107+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:59.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:59.107+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:59.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:59.107+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:13:59.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:13:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:13:59.108+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:13:59.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:59.022+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:59.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:59.022+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:59.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:59.022+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:13:59.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:13:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:13:59.023+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:00.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:00.111+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:00.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:00.111+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:00.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:00.111+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:00.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:00.112+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:00.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:00.043+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:00.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:00.043+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:00.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:00.043+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:00.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:00.044+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:00.341 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:00.142+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:00.341 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:00.142+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:00.341 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:00.142+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:00.341 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:00.142+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:00.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:14:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:14:00.339+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19494 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:14:01.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:01.118+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:01.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:01.118+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:01.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:01.118+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:01.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:01.118+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:01.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:01.116+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:01.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:01.116+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:01.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:01.116+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:01.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:01.117+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:01.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:01.062+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:01.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:01.062+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:01.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:01.062+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:01.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:01.062+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:02.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:02.160+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:02.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:02.160+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:02.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:02.160+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:02.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:02.161+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:02.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:02.102+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:02.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:02.102+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:02.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:02.102+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:02.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:02.103+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:02.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:02.105+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:02.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:02.105+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:02.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:02.105+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:02.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:02.106+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:03.156 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:03.099+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:03.156 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:03.099+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:03.156 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:03.099+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:03.157 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:03.099+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:03.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:03.202+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:03.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:03.202+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:03.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:03.202+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:03.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:03.203+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:03.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:03.153+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:03.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:03.154+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:03.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:03.154+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:03.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:03.155+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:04.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:04.116+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:04.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:04.116+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:04.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:04.116+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:04.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:04.117+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:04.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:04.189+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:04.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:04.189+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:04.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:04.189+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:04.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:04.190+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:04.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:04.162+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:04.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:04.162+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:04.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:04.162+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:04.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:04.163+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:05.466 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:14:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:14:05.341+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19579 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:14:05.466 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:05.238+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:05.466 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:05.238+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:05.467 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:05.238+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:05.467 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:05.238+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:05.467 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:05.140+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:05.467 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:05.140+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:05.467 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:05.140+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:05.468 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:05.141+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:05.468 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:05.155+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:05.468 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:05.155+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:05.468 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:05.155+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:05.468 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:05.156+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:06.211 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:06.189+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:06.211 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:06.189+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:06.211 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:06.189+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:06.211 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:06.191+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:06.212 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:06.117+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:06.212 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:06.117+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:06.212 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:06.117+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:06.212 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:06.117+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:06.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:06.212+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:06.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:06.212+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:06.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:06.212+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:06.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:06.213+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:07.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:07.138+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:07.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:07.138+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:07.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:07.138+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:07.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:07.139+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:07.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:07.189+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:07.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:07.189+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:07.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:07.189+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:07.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:07.190+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:07.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:07.159+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:07.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:07.159+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:07.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:07.159+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:07.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:07.159+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:08.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:08.176+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:08.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:08.176+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:08.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:08.176+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:08.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:08.177+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:08.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:08.213+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:08.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:08.213+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:08.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:08.213+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:08.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:08.214+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:08.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:08.113+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:08.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:08.113+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:08.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:08.113+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:08.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:08.114+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:09.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:09.212+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:09.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:09.212+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:09.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:09.212+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:09.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:09.212+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:09.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:09.150+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:09.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:09.150+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:09.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:09.150+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:09.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:09.151+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:09.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:09.206+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:09.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:09.206+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:09.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:09.206+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:09.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:09.207+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:10.466 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:14:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:14:10.343+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19663 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:14:10.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:10.256+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:10.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:10.256+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:10.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:10.256+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:10.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:10.257+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:10.468 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:10.177+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:10.468 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:10.177+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:10.468 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:10.177+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:10.468 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:10.178+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:10.468 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:10.173+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:10.468 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:10.173+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:10.469 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:10.173+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:10.469 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:10.173+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:11.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:11.300+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:11.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:11.300+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:11.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:11.300+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:11.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:11.301+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:11.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:11.183+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:11.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:11.183+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:11.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:11.183+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:11.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:11.184+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:11.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:11.207+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:11.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:11.207+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:11.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:11.207+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:11.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:11.208+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:12.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:12.336+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:12.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:12.336+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:12.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:12.336+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:12.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:12.337+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:12.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:12.186+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:12.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:12.186+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:12.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:12.186+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:12.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:12.187+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:12.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:12.170+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:12.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:12.170+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:12.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:12.170+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:12.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:12.170+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:13.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:13.309+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:13.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:13.309+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:13.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:13.309+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:13.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:13.309+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:13.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:13.182+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:13.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:13.182+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:13.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:13.182+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:13.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:13.182+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:13.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:13.179+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:13.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:13.179+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:13.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:13.179+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:13.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:13.179+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:14.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:14.319+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:14.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:14.319+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:14.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:14.319+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:14.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:14.320+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:14.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:14.212+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:14.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:14.212+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:14.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:14.212+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:14.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:14.213+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:14.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:14.150+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:14.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:14.150+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:14.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:14.150+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:14.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:14.151+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:15.369 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:14:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:14:15.345+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19748 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:14:15.370 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:15.283+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:15.370 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:15.283+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:15.370 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:15.283+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:15.370 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:15.284+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:15.370 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:15.118+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:15.371 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:15.118+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:15.371 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:15.118+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:15.371 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:15.119+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:15.371 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:15.222+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:15.371 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:15.222+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:15.371 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:15.222+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:15.372 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:15.223+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:16.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:16.295+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:16.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:16.295+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:16.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:16.295+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:16.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:16.296+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:16.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:16.152+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:16.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:16.152+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:16.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:16.152+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:16.477 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:16.153+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:16.477 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:16.175+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:16.477 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:16.175+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:16.477 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:16.175+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:16.477 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:16.176+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:17.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:17.295+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:17.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:17.295+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:17.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:17.295+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:17.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:17.296+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:17.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:17.199+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:17.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:17.199+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:17.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:17.199+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:17.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:17.199+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:17.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:17.175+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:17.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:17.175+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:17.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:17.175+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:17.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:17.175+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:18.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:18.211+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:18.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:18.211+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:18.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:18.211+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:18.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:18.212+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:18.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:18.186+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:18.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:18.186+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:18.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:18.186+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:18.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:18.187+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:18.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:18.278+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:18.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:18.278+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:18.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:18.278+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:18.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:18.279+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:19.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:19.323+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:19.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:19.323+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:19.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:19.323+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:19.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:19.324+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:19.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:19.248+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:19.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:19.248+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:19.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:19.248+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:19.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:19.249+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:19.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:19.213+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:19.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:19.213+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:19.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:19.213+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:19.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:19.213+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:20.469 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:14:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:14:20.347+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19832 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:14:20.469 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:20.258+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:20.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:20.258+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:20.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:20.258+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:20.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:20.259+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:20.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:20.228+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:20.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:20.228+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:20.470 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:20.228+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:20.471 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:20.229+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:20.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:20.372+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:20.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:20.372+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:20.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:20.372+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:20.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:20.374+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:21.711 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:21.290+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:21.711 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:21.290+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:21.711 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:21.290+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:21.711 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:21.290+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:21.711 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:21.331+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:21.712 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:21.331+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:21.712 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:21.331+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:21.712 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:21.331+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:21.712 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:21.249+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:21.712 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:21.249+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:21.712 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:21.249+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:21.713 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:21.250+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:22.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:22.297+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:22.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:22.297+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:22.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:22.297+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:22.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:22.298+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:22.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:22.353+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:22.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:22.353+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:22.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:22.353+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:22.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:22.354+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:22.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:22.268+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:22.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:22.268+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:22.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:22.268+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:22.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:22.268+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:23.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:23.335+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:23.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:23.335+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:23.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:23.335+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:23.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:23.336+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:23.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:23.341+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:23.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:23.341+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:23.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:23.341+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:23.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:23.342+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:23.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:23.296+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:23.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:23.296+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:23.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:23.297+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:23.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:23.298+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:24.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:24.327+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:24.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:24.327+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:24.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:24.327+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:24.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:24.328+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:24.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:24.292+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:24.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:24.292+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:24.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:24.292+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:24.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:24.293+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:24.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:24.299+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:24.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:24.299+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:24.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:24.299+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:24.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:24.300+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:25.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:14:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:14:25.349+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19917 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:14:25.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:25.288+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:25.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:25.288+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:25.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:25.288+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:25.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:25.289+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:25.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:25.305+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:25.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:25.305+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:25.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:25.305+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:25.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:25.306+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:25.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:25.335+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:25.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:25.335+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:25.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:25.335+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:25.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:25.336+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:26.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:26.306+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:26.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:26.306+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:26.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:26.306+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:26.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:26.307+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:26.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:26.306+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:26.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:26.306+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:26.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:26.306+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:26.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:26.306+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:26.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:26.338+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:26.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:26.338+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:26.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:26.338+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:26.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:26.338+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:27.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:27.295+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:27.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:27.295+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:27.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:27.295+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:27.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:27.296+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:27.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:27.283+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:27.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:27.283+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:27.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:27.283+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:27.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:27.284+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:27.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:27.348+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:27.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:27.348+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:27.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:27.348+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:27.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:27.348+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:28.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:28.293+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:28.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:28.293+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:28.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:28.293+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:28.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:28.294+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:28.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:28.353+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:28.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:28.353+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:28.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:28.353+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:28.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:28.354+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:28.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:28.303+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:28.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:28.303+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:28.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:28.303+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:28.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:28.304+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:29.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:29.321+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:29.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:29.321+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:29.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:29.321+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:29.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:29.322+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:29.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:29.297+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:29.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:29.297+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:29.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:29.297+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:29.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:29.298+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:29.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:29.323+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:29.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:29.323+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:29.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:29.323+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:29.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:29.324+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:30.473 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:14:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:14:30.351+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20001 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:14:30.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:30.345+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:30.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:30.345+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:30.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:30.345+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:30.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:30.346+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:30.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:30.338+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:30.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:30.338+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:30.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:30.338+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:30.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:30.339+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:30.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:30.294+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:30.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:30.294+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:30.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:30.294+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:30.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:30.295+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:31.711 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:31.371+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:31.711 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:31.371+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:31.711 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:31.371+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:31.711 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:31.371+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:31.712 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:31.342+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:31.712 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:31.342+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:31.712 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:31.342+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:31.712 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:31.343+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:31.712 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:31.326+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:31.713 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:31.326+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:31.713 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:31.326+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:31.713 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:31.327+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:32.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:32.335+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:32.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:32.335+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:32.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:32.335+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:32.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:32.335+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:32.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:32.385+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:32.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:32.385+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:32.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:32.385+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:32.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:32.385+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:32.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:32.307+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:32.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:32.307+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:32.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:32.307+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:32.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:32.307+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:33.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:33.360+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:33.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:33.360+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:33.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:33.360+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:33.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:33.361+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:33.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:33.369+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:33.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:33.369+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:33.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:33.369+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:33.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:33.370+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:33.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:33.293+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:33.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:33.293+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:33.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:33.293+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:33.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:33.293+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:34.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:34.320+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:34.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:34.320+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:34.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:34.320+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:34.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:34.321+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:34.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:34.348+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:34.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:34.348+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:34.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:34.348+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:34.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:34.349+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:34.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:34.336+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:34.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:34.336+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:34.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:34.336+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:34.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:34.337+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:35.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:14:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:14:35.353+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20086 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:14:35.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:35.340+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:35.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:35.340+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:35.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:35.340+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:35.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:35.340+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:35.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:35.369+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:35.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:35.369+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:35.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:35.369+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:35.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:35.369+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:35.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:35.355+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:35.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:35.355+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:35.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:35.355+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:35.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:35.356+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:36.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:36.383+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:36.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:36.383+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:36.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:36.383+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:36.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:36.384+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:36.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:36.368+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:36.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:36.368+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:36.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:36.369+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:36.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:36.370+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:36.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:36.333+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:36.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:36.333+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:36.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:36.333+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:36.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:36.334+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:37.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:37.420+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:37.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:37.420+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:37.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:37.420+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:37.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:37.421+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:37.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:37.356+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:37.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:37.356+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:37.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:37.356+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:37.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:37.356+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:37.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:37.351+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:37.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:37.351+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:37.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:37.351+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:37.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:37.352+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:38.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:38.410+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:38.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:38.410+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:38.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:38.410+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:38.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:38.410+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:38.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:38.319+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:38.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:38.319+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:38.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:38.319+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:38.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:38.320+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:38.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:38.319+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:38.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:38.319+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:38.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:38.319+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:38.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:38.320+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:39.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:39.448+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:39.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:39.448+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:39.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:39.448+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:39.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:39.449+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:39.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:39.270+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:39.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:39.270+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:39.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:39.270+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:39.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:39.270+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:39.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:39.340+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:39.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:39.340+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:39.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:39.340+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:39.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:39.340+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:40.440 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:14:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:14:40.354+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20170 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:14:40.441 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:40.334+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:40.441 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:40.334+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:40.441 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:40.334+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:40.441 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:40.335+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:40.442 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:40.259+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:40.442 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:40.259+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:40.442 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:40.259+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:40.442 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:40.259+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:40.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:40.440+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:40.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:40.440+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:40.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:40.440+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:40.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:40.441+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:41.711 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:41.360+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:41.711 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:41.360+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:41.712 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:41.360+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:41.712 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:41.361+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:41.712 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:41.458+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:41.712 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:41.458+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:41.713 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:41.458+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:41.713 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:41.459+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:41.713 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:41.283+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:41.713 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:41.283+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:41.713 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:41.283+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:41.713 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:41.284+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:42.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:42.364+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:42.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:42.364+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:42.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:42.364+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:42.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:42.365+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:42.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:42.507+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:42.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:42.507+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:42.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:42.507+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:42.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:42.508+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:42.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:42.314+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:42.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:42.314+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:42.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:42.314+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:42.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:42.315+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:43.547 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:43.351+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:43.547 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:43.351+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:43.547 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:43.351+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:43.547 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:43.352+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:43.548 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:43.268+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:43.548 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:43.268+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:43.548 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:43.268+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:43.548 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:43.268+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:43.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:43.546+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:43.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:43.546+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:43.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:43.546+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:43.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:43.547+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:44.591 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:44.327+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:44.592 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:44.327+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:44.592 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:44.327+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:44.592 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:44.328+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:44.592 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:44.267+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:44.592 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:44.267+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:44.592 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:44.267+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:44.593 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:44.268+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:44.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:44.591+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:44.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:44.591+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:44.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:44.591+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:44.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:44.592+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:45.591 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:14:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:14:45.356+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20255 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:14:45.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:45.303+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:45.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:45.303+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:45.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:45.303+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:45.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:45.303+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:45.592 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:45.348+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:45.592 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:45.348+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:45.592 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:45.348+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:45.592 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:45.349+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:45.893 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:45.590+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:45.894 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:45.590+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:45.894 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:45.590+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:45.894 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:45.591+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:46.627 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:46.353+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:46.628 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:46.353+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:46.628 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:46.353+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:46.628 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:46.354+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:46.628 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:46.294+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:46.629 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:46.294+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:46.629 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:46.294+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:46.629 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:46.295+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:46.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:46.627+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:46.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:46.627+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:46.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:46.627+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:46.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:46.628+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:47.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:47.585+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:47.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:47.585+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:47.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:47.585+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:47.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:47.586+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:47.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:47.402+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:47.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:47.402+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:47.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:47.402+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:47.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:47.403+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:47.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:47.342+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:47.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:47.342+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:47.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:47.342+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:47.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:47.342+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:48.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:48.543+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:48.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:48.543+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:48.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:48.543+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:48.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:48.543+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:48.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:48.401+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:48.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:48.401+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:48.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:48.401+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:48.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:48.402+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:48.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:48.384+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:48.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:48.384+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:48.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:48.384+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:48.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:48.385+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:49.509 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:49.404+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:49.510 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:49.404+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:49.510 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:49.404+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:49.510 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:49.405+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:49.510 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:49.341+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:49.510 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:49.341+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:49.510 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:49.341+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:49.511 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:49.341+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:49.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:49.509+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:49.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:49.509+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:49.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:49.509+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:49.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:49.510+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:50.453 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:14:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:14:50.358+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20339 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:14:50.453 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:50.316+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:50.453 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:50.316+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:50.453 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:50.316+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:50.453 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:50.317+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:50.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:50.490+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:50.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:50.491+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:50.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:50.491+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:50.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:50.491+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:50.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:50.452+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:50.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:50.452+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:50.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:50.452+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:50.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:50.453+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:51.711 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:51.465+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:51.711 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:51.465+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:51.712 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:51.465+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:51.712 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:51.466+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:51.712 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:51.453+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:51.712 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:51.453+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:51.712 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:51.453+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:51.713 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:51.453+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:51.713 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:51.329+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:51.713 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:51.329+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:51.713 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:51.329+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:51.713 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:51.330+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:52.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:52.495+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:52.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:52.495+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:52.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:52.495+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:52.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:52.496+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:52.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:52.359+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:52.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:52.359+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:52.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:52.359+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:52.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:52.360+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:52.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:52.486+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:52.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:52.486+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:52.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:52.486+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:52.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:52.487+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:53.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:53.512+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:53.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:53.512+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:53.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:53.512+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:53.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:53.512+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:53.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:53.522+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:53.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:53.522+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:53.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:53.522+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:53.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:53.523+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:53.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:53.322+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:53.592 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:53.322+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:53.592 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:53.322+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:53.592 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:53.323+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:54.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:54.549+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:54.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:54.549+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:54.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:54.549+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:54.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:54.550+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:54.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:54.550+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:54.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:54.550+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:54.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:54.550+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:54.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:54.551+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:54.314+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:54.314+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:54.314+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:54.314+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:55.361 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:55.350+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:55.361 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:55.350+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:55.362 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:55.350+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:55.362 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:55.351+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:55.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:14:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:14:55.360+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20421 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:14:55.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:55.592+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:55.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:55.592+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:55.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:55.592+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:55.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:55.592+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:55.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:55.527+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:55.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:55.527+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:55.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:55.527+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:55.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:55.527+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:56.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:56.552+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:56.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:56.552+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:56.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:56.552+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:56.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:56.552+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:56.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:56.553+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:56.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:56.553+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:56.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:56.553+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:56.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:56.554+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:56.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:56.396+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:56.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:56.396+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:56.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:56.396+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:56.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:56.397+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:57.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:57.591+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:57.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:57.591+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:57.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:57.591+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:57.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:57.591+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:57.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:57.438+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:57.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:57.438+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:57.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:57.438+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:57.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:57.438+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:57.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:57.532+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:57.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:57.532+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:57.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:57.532+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:57.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:57.533+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:58.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:58.534+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:58.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:58.534+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:58.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:58.534+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:58.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:58.534+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:14:58.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:58.584+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:58.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:58.584+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:58.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:58.584+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:58.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:58.584+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:58.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:58.470+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:58.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:58.470+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:58.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:58.470+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:58.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:58.471+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:59.537 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:59.458+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:59.537 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:59.458+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:59.537 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:59.458+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:14:59.538 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:14:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:14:59.459+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:59.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:59.538+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:59.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:59.538+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:59.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:59.538+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:14:59.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:14:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:14:59.538+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:14:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:59.542+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:59.543+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:59.543+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:14:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:14:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:14:59.543+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:00.454 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:15:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:15:00.364+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20502 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:15:00.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:00.502+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:00.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:00.502+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:00.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:00.502+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:00.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:00.503+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:00.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:00.577+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:00.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:00.577+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:00.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:00.577+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:00.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:00.577+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:00.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:00.455+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:00.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:00.455+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:00.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:00.455+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:00.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:00.456+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:01.545 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:01.490+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:01.546 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:01.490+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:01.546 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:01.490+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:01.546 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:01.491+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:01.546 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:01.429+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:01.546 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:01.429+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:01.546 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:01.429+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:01.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:01.430+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:01.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:01.546+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:01.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:01.546+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:01.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:01.546+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:01.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:01.547+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:02.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:02.484+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:02.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:02.484+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:02.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:02.484+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:02.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:02.485+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:02.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:02.529+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:02.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:02.529+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:02.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:02.529+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:02.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:02.530+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:02.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:02.432+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:02.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:02.433+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:02.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:02.433+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:02.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:02.433+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:03.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:03.473+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:03.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:03.473+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:03.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:03.473+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:03.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:03.474+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:03.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:03.440+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:03.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:03.440+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:03.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:03.440+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:03.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:03.441+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:03.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:03.556+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:03.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:03.556+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:03.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:03.556+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:03.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:03.557+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:04.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:04.470+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:04.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:04.470+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:04.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:04.470+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:04.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:04.471+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:04.592 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:04.472+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:04.592 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:04.472+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:04.592 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:04.472+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:04.592 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:04.473+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:04.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:04.591+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:04.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:04.591+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:04.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:04.591+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:04.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:04.592+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:05.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:15:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:15:05.366+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20581 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:15:05.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:05.506+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:05.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:05.506+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:05.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:05.506+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:05.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:05.507+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:05.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:05.578+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:05.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:05.578+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:05.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:05.578+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:05.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:05.579+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:05.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:05.440+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:05.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:05.440+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:05.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:05.440+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:05.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:05.441+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:06.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:06.478+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:06.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:06.478+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:06.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:06.478+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:06.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:06.478+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:06.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:06.580+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:06.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:06.580+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:06.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:06.580+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:06.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:06.581+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:06.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:06.454+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:06.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:06.454+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:06.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:06.454+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:06.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:06.455+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:07.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:07.516+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:07.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:07.516+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:07.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:07.516+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:07.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:07.516+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:07.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:07.532+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:07.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:07.532+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:07.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:07.532+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:07.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:07.532+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:07.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:07.429+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:07.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:07.429+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:07.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:07.429+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:07.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:07.429+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:08.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:08.483+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:08.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:08.483+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:08.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:08.483+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:08.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:08.483+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:08.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:08.493+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:08.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:08.493+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:08.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:08.493+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:08.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:08.493+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:08.727 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:08.440+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:08.727 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:08.440+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:08.727 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:08.440+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:08.728 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:08.441+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:09.521 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:09.448+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:09.521 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:09.448+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:09.521 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:09.448+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:09.521 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:09.448+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:09.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:09.537+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:09.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:09.537+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:09.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:09.537+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:09.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:09.538+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:09.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:09.521+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:09.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:09.521+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:09.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:09.521+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:09.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:09.522+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:10.464 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:15:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:15:10.368+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20659 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:15:10.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:10.546+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:10.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:10.546+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:10.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:10.546+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:10.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:10.547+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:10.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:10.488+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:10.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:10.488+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:10.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:10.488+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:10.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:10.489+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:10.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:10.465+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:10.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:10.465+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:10.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:10.465+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:10.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:10.465+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:11.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:11.490+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:11.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:11.490+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:11.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:11.490+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:11.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:11.491+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:11.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:11.470+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:11.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:11.470+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:11.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:11.470+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:11.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:11.470+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:11.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:11.569+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:11.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:11.569+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:11.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:11.569+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:11.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:11.570+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:12.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:12.570+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:12.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:12.570+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:12.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:12.570+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:12.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:12.571+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:12.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:12.523+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:12.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:12.523+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:12.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:12.523+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:12.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:12.523+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:12.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:12.435+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:12.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:12.435+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:12.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:12.435+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:12.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:12.436+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:13.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:13.498+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:13.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:13.498+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:13.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:13.498+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:13.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:13.499+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:13.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:13.536+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:13.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:13.536+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:13.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:13.536+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:13.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:13.537+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:13.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:13.472+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:13.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:13.472+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:13.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:13.472+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:13.727 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:13.473+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:14.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:14.463+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:14.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:14.463+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:14.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:14.463+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:14.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:14.464+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:14.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:14.483+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:14.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:14.483+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:14.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:14.483+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:14.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:14.484+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:14.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:14.506+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:14.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:14.506+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:14.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:14.506+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:14.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:14.506+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:15.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:15:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:15:15.370+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20738 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:15:15.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:15.525+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:15.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:15.525+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:15.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:15.525+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:15.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:15.526+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:15.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:15.445+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:15.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:15.445+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:15.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:15.445+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:15.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:15.445+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:15.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:15.504+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:15.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:15.504+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:15.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:15.504+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:15.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:15.505+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:16.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:16.548+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:16.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:16.548+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:16.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:16.548+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:16.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:16.549+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:16.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:16.501+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:16.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:16.501+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:16.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:16.501+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:16.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:16.502+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:16.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:16.482+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:16.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:16.482+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:16.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:16.482+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:16.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:16.483+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:17.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:17.508+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:17.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:17.508+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:17.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:17.508+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:17.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:17.509+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:17.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:17.467+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:17.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:17.467+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:17.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:17.467+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:17.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:17.468+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:17.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:17.513+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:17.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:17.513+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:17.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:17.513+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:17.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:17.514+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:18.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:18.479+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:18.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:18.479+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:18.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:18.479+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:18.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:18.480+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:18.977 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:18.476+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:18.977 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:18.476+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:18.977 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:18.476+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:18.977 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:18.477+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:18.978 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:18.502+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:18.978 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:18.502+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:18.978 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:18.502+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:18.978 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:18.503+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:19.506 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:19.433+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:19.506 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:19.433+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:19.506 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:19.433+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:19.507 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:19.434+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:19.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:19.514+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:19.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:19.514+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:19.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:19.514+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:19.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:19.515+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:19.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:19.507+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:19.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:19.507+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:19.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:19.507+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:19.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:19.508+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:20.473 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:15:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:15:20.372+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20816 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:15:20.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:20.415+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:20.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:20.415+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:20.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:20.415+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:20.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:20.417+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:20.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:20.479+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:20.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:20.479+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:20.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:20.479+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:20.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:20.480+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:20.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:20.475+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:20.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:20.475+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:20.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:20.475+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:20.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:20.476+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:21.713 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:21.448+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:21.713 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:21.448+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:21.713 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:21.448+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:21.713 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:21.449+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:21.714 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:21.489+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:21.714 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:21.489+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:21.714 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:21.489+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:21.714 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:21.490+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:21.714 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:21.433+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:21.714 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:21.433+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:21.715 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:21.433+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:21.715 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:21.434+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:22.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:22.424+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:22.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:22.424+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:22.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:22.424+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:22.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:22.424+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:22.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:22.470+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:22.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:22.470+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:22.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:22.470+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:22.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:22.471+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:22.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:22.485+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:22.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:22.485+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:22.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:22.485+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:22.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:22.485+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:23.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:23.396+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:23.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:23.396+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:23.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:23.396+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:23.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:23.397+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:23.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:23.467+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:23.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:23.467+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:23.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:23.467+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:23.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:23.469+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:23.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:23.468+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:23.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:23.468+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:23.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:23.468+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:23.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:23.469+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:24.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:24.405+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:24.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:24.405+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:24.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:24.405+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:24.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:24.405+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:24.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:24.435+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:24.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:24.435+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:24.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:24.435+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:24.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:24.435+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:24.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:24.505+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:24.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:24.505+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:24.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:24.505+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:24.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:24.506+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:25.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:15:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:15:25.374+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20898 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:15:25.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:25.409+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:25.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:25.409+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:25.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:25.410+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:25.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:25.410+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:25.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:25.452+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:25.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:25.452+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:25.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:25.452+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:25.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:25.453+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:25.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:25.505+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:25.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:25.505+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:25.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:25.505+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:25.726 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:25.506+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:26.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:26.404+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:26.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:26.405+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:26.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:26.405+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:26.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:26.405+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:26.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:26.447+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:26.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:26.447+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:26.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:26.447+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:26.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:26.447+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:26.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:26.556+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:26.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:26.556+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:26.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:26.556+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:26.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:26.557+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:27.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:27.456+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:27.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:27.456+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:27.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:27.456+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:27.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:27.456+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:27.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:27.537+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:27.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:27.537+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:27.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:27.537+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:27.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:27.538+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:27.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:27.380+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:27.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:27.380+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:27.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:27.380+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:27.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:27.381+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:28.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:28.448+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:28.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:28.448+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:28.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:28.448+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:28.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:28.448+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:28.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:28.577+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:28.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:28.577+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:28.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:28.577+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:28.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:28.578+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:28.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:28.393+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:28.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:28.393+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:28.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:28.393+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:28.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:28.394+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:29.416+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:29.416+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:29.416+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:29.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:29.417+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:29.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:29.351+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:29.351+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:29.351+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:29.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:29.352+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:29.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:29.588+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:29.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:29.588+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:29.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:29.588+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:29.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:29.589+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:30.437 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:15:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:15:30.376+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20978 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:15:30.437 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:30.352+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:30.437 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:30.352+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:30.437 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:30.352+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:30.438 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:30.352+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:30.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:30.437+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:30.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:30.437+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:30.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:30.437+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:30.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:30.438+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:30.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:30.591+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:30.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:30.591+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:30.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:30.591+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:30.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:30.591+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:31.577 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:31.308+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:31.577 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:31.308+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:31.577 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:31.308+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:31.577 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:31.309+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:31.577 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:31.473+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:31.578 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:31.473+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:31.578 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:31.474+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:31.578 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:31.474+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:31.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:31.577+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:31.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:31.577+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:31.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:31.577+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:31.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:31.578+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:32.595 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:32.334+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:32.595 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:32.334+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:32.596 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:32.334+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:32.596 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:32.335+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:32.596 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:32.473+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:32.596 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:32.473+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:32.596 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:32.473+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:32.596 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:32.473+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:32.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:32.596+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:32.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:32.596+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:32.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:32.596+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:32.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:32.596+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:33.592 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:33.517+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:33.592 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:33.517+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:33.592 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:33.517+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:33.592 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:33.517+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:33.593 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:33.333+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:33.593 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:33.333+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:33.593 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:33.333+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:33.593 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:33.333+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:33.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:33.633+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:33.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:33.633+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:33.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:33.633+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:33.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:33.634+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:34.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:34.623+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:34.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:34.623+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:34.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:34.623+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:34.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:34.624+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:34.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:34.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:34.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:34.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:34.727 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:34.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:34.727 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:34.377+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:34.728 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:34.495+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:34.728 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:34.495+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:34.728 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:34.495+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:34.728 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:34.496+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:35.660 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:15:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:15:35.379+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21064 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:15:35.661 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:35.467+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:35.661 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:35.467+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:35.661 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:35.467+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:35.661 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:35.468+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:35.662 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:35.356+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:35.662 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:35.356+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:35.662 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:35.356+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:35.662 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:35.356+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:35.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:35.661+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:35.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:35.661+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:35.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:35.661+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:35.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:35.662+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:36.619 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:36.511+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:36.619 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:36.511+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:36.619 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:36.511+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:36.620 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:36.511+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:36.620 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:36.317+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:36.620 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:36.317+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:36.620 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:36.317+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:36.620 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:36.317+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:36.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:36.620+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:36.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:36.620+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:36.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:36.620+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:36.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:36.620+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:37.638 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:37.489+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:37.638 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:37.489+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:37.638 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:37.489+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:37.639 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:37.490+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:37.639 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:37.348+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:37.639 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:37.348+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:37.639 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:37.348+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:37.639 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:37.349+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:37.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:37.638+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:37.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:37.638+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:37.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:37.638+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:37.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:37.639+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:38.687 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:38.447+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:38.687 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:38.447+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:38.687 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:38.447+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:38.687 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:38.448+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:38.688 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:38.310+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:38.688 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:38.310+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:38.688 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:38.310+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:38.688 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:38.311+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:38.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:38.687+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:38.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:38.687+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:38.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:38.687+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:38.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:38.688+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:39.716 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:39.354+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:39.716 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:39.354+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:39.716 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:39.354+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:39.717 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:39.354+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:39.717 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:39.448+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:39.717 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:39.448+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:39.717 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:39.448+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:39.717 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:39.449+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:39.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:39.716+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:39.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:39.716+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:39.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:39.716+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:39.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:39.717+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:40.473 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:15:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:15:40.381+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21147 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:15:40.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:40.384+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:40.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:40.384+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:40.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:40.384+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:40.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:40.385+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:40.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:40.403+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:40.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:40.403+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:40.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:40.403+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:40.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:40.403+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:40.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:40.705+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:40.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:40.705+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:40.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:40.705+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:40.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:40.705+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:41.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:41.431+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:41.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:41.431+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:41.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:41.431+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:41.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:41.432+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:41.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:41.385+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:41.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:41.385+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:41.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:41.385+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:41.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:41.386+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:41.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:41.730+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:41.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:41.730+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:41.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:41.730+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:41.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:41.731+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:42.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:42.426+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:42.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:42.426+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:42.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:42.426+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:42.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:42.427+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:42.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:42.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:42.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:42.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:42.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:42.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:42.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:42.377+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:43.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:42.768+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:43.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:42.768+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:43.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:42.768+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:43.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:42.768+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:43.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:43.462+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:43.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:43.462+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:43.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:43.462+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:43.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:43.463+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:43.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:43.348+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:43.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:43.348+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:43.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:43.348+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:43.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:43.348+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:44.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:43.741+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:44.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:43.741+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:44.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:43.741+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:44.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:43.742+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:44.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:44.501+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:44.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:44.501+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:44.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:44.501+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:44.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:44.502+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:44.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:44.301+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:44.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:44.301+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:44.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:44.301+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:44.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:44.302+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:44.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:44.699+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:44.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:44.699+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:44.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:44.699+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:44.977 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:44.700+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:45.548 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:15:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:15:45.383+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21233 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:15:45.548 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:45.263+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:45.549 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:45.263+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:45.549 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:45.263+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:45.549 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:45.264+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:45.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:45.548+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:45.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:45.548+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:45.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:45.548+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:45.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:45.549+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:45.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:45.719+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:45.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:45.719+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:45.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:45.719+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:45.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:45.720+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:46.585 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:46.229+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:46.585 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:46.229+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:46.585 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:46.229+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:46.585 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:46.229+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:46.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:46.585+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:46.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:46.585+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:46.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:46.585+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:46.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:46.586+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:46.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:46.744+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:46.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:46.744+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:46.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:46.744+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:46.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:46.745+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:47.597 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:47.242+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:47.597 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:47.242+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:47.597 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:47.242+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:47.597 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:47.243+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:47.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:47.728+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:47.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:47.728+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:47.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:47.728+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:47.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:47.729+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:47.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:47.597+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:47.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:47.597+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:47.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:47.597+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:47.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:47.598+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:48.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:48.211+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:48.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:48.211+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:48.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:48.211+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:48.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:48.211+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:48.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:48.682+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:48.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:48.682+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:48.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:48.682+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:48.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:48.683+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:48.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:48.636+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:48.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:48.636+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:48.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:48.636+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:48.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:48.637+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:49.676 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:49.259+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:49.676 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:49.259+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:49.676 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:49.259+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:49.676 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:49.259+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:49.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:49.676+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:49.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:49.676+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:49.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:49.676+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:49.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:49.677+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:49.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:49.676+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:49.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:49.676+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:49.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:49.676+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:49.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:49.677+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:50.473 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:15:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:15:50.385+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21316 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:15:50.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:50.218+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:50.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:50.218+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:50.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:50.218+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:50.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:50.219+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:50.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:50.633+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:50.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:50.633+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:50.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:50.633+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:50.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:50.634+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:50.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:50.693+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:50.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:50.693+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:50.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:50.693+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:50.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:50.693+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:51.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:51.172+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:51.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:51.172+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:51.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:51.172+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:51.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:51.173+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:51.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:51.645+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:51.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:51.645+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:51.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:51.645+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:51.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:51.645+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:51.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:51.705+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:51.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:51.705+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:51.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:51.705+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:51.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:51.706+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:52.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:52.195+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:52.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:52.195+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:52.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:52.195+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:52.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:52.195+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:52.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:52.665+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:52.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:52.665+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:52.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:52.665+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:52.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:52.666+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:52.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:52.690+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:52.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:52.690+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:52.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:52.690+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:52.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:52.691+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:53.594 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:53.244+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:53.594 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:53.244+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:53.594 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:53.244+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:53.595 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:53.245+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:53.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:53.714+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:53.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:53.714+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:53.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:53.714+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:53.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:53.714+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:53.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:53.704+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:53.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:53.704+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:53.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:53.704+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:53.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:53.705+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:54.717 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:54.255+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:54.718 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:54.255+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:54.718 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:54.255+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:54.718 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:54.255+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:54.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:54.761+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:54.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:54.761+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:54.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:54.761+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:54.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:54.762+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:54.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:54.718+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:54.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:54.718+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:54.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:54.718+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:54.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:54.719+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:55.512 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:15:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:15:55.387+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21402 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:15:55.513 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:55.238+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:55.513 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:55.238+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:55.513 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:55.238+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:55.513 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:55.239+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:55.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:55.733+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:55.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:55.733+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:55.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:55.733+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:55.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:55.734+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:55.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:55.689+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:55.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:55.689+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:55.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:55.689+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:55.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:55.689+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:56.549 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:56.260+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:56.550 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:56.260+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:56.550 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:56.260+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:56.550 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:56.260+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:56.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:56.774+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:56.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:56.774+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:56.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:56.774+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:56.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:56.775+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:56.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:56.730+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:56.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:56.730+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:56.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:56.730+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:56.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:56.730+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:57.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:57.296+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:57.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:57.296+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:57.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:57.296+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:57.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:57.297+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:58.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:57.756+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:58.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:57.756+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:58.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:57.756+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:58.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:57.757+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:58.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:57.741+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:58.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:57.741+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:58.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:57.741+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:58.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:57.742+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:58.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:58.269+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:58.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:58.269+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:58.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:58.269+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:58.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:58.270+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:59.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:58.769+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:59.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:58.769+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:59.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:58.769+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:59.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:58.770+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:59.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:58.740+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:59.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:58.740+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:59.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:58.740+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:59.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:58.741+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:15:59.695 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:59.236+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:59.695 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:59.236+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:59.695 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:59.236+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:15:59.695 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:15:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:15:59.237+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:59.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:59.782+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:59.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:59.782+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:59.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:59.782+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:15:59.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:15:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:15:59.783+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:15:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:59.695+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:59.695+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:59.695+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:15:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:15:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:15:59.696+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:00.473 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:16:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:16:00.388+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21485 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:16:00.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:00.226+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:00.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:00.226+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:00.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:00.226+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:00.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:00.226+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:00.816 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:00.692+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:00.816 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:00.692+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:00.816 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:00.692+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:00.817 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:00.692+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:01.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:00.816+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:01.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:00.816+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:01.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:00.816+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:01.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:00.817+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:01.661 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:01.249+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:01.662 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:01.249+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:01.662 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:01.249+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:01.662 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:01.249+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:01.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:01.661+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:01.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:01.661+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:01.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:01.661+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:01.977 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:01.662+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:01.977 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:01.803+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:01.977 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:01.803+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:01.977 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:01.803+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:01.978 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:01.804+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:02.693 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:02.233+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:02.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:02.233+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:02.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:02.233+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:02.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:02.234+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:02.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:02.693+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:02.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:02.693+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:02.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:02.693+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:02.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:02.694+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:02.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:02.843+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:02.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:02.843+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:02.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:02.843+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:02.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:02.844+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:03.594 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:03.264+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:03.595 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:03.264+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:03.595 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:03.264+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:03.595 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:03.265+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:03.854 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:03.694+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:03.855 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:03.694+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:03.855 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:03.694+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:03.855 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:03.695+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:04.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:03.854+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:04.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:03.854+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:04.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:03.854+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:04.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:03.854+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:04.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:04.250+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:04.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:04.250+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:04.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:04.250+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:04.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:04.251+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:05.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:04.902+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:05.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:04.902+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:05.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:04.902+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:05.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:04.903+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:05.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:04.742+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:05.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:04.742+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:05.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:04.742+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:05.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:04.743+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:05.519 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:16:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:16:05.390+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21570 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:16:05.520 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:05.266+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:05.520 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:05.266+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:05.520 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:05.266+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:05.520 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:05.266+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:06.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:05.792+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:06.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:05.792+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:06.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:05.792+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:06.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:05.793+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:06.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:05.934+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:06.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:05.934+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:06.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:05.934+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:06.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:05.935+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:06.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:06.247+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:06.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:06.247+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:06.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:06.247+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:06.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:06.247+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:07.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:06.960+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:07.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:06.960+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:07.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:06.960+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:07.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:06.960+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:07.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:06.753+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:07.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:06.753+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:07.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:06.753+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:07.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:06.754+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:07.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:07.237+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:07.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:07.237+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:07.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:07.237+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:07.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:07.238+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:08.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:07.973+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:08.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:07.973+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:08.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:07.973+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:08.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:07.973+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:08.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:07.757+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:08.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:07.757+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:08.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:07.757+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:08.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:07.757+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:08.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:08.287+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:08.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:08.287+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:08.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:08.287+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:08.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:08.288+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:09.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:08.959+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:09.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:08.959+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:09.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:08.959+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:09.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:08.959+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:09.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:08.745+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:09.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:08.745+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:09.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:08.745+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:09.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:08.746+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:09.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:09.264+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:09.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:09.264+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:09.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:09.264+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:09.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:09.265+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:10.184 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:09.746+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:10.184 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:09.746+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:10.184 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:09.746+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:10.185 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:09.746+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:10.185 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:09.960+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:10.185 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:09.960+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:10.185 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:09.960+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:10.185 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:09.961+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:10.473 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:16:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:16:10.393+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21653 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:16:10.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:10.243+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:10.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:10.243+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:10.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:10.243+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:10.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:10.243+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:10.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:10.736+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:10.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:10.736+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:10.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:10.736+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:10.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:10.736+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:11.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:11.009+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:11.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:11.009+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:11.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:11.009+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:11.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:11.010+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:11.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:11.234+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:11.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:11.234+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:11.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:11.234+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:11.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:11.235+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:11.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:11.727+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:11.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:11.727+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:11.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:11.727+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:11.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:11.727+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:12.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:12.049+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:12.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:12.049+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:12.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:12.049+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:12.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:12.050+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:12.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:12.257+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:12.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:12.257+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:12.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:12.257+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:12.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:12.258+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:12.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:12.709+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:12.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:12.709+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:12.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:12.709+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:12.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:12.710+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:13.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:13.282+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:13.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:13.282+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:13.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:13.282+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:13.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:13.283+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:13.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:13.097+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:13.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:13.097+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:13.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:13.097+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:13.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:13.098+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:13.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:13.703+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:13.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:13.703+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:13.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:13.703+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:13.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:13.705+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:14.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:14.329+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:14.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:14.329+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:14.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:14.329+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:14.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:14.330+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:14.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:14.121+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:14.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:14.121+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:14.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:14.121+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:14.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:14.122+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:14.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:14.713+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:14.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:14.713+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:14.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:14.713+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:14.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:14.713+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:15.313 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:15.103+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:15.313 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:15.103+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:15.313 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:15.103+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:15.313 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:15.104+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:15.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:16:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:16:15.395+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21739 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:16:15.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:15.312+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:15.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:15.312+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:15.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:15.312+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:15.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:15.313+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:16.062 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:15.744+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:16.062 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:15.744+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:16.062 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:15.744+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:16.062 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:15.745+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:16.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:16.061+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:16.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:16.061+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:16.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:16.061+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:16.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:16.062+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:16.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:16.279+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:16.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:16.279+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:16.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:16.279+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:16.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:16.280+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:17.109 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:16.758+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:17.109 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:16.758+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:17.109 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:16.758+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:17.109 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:16.758+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:17.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:17.108+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:17.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:17.108+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:17.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:17.108+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:17.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:17.109+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:17.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:17.290+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:17.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:17.290+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:17.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:17.290+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:17.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:17.290+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:18.102 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:17.804+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:18.102 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:17.804+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:18.102 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:17.804+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:18.102 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:17.805+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:18.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:18.300+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:18.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:18.300+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:18.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:18.300+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:18.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:18.301+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:18.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:18.101+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:18.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:18.101+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:18.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:18.101+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:18.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:18.102+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:19.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:18.806+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:19.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:18.806+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:19.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:18.806+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:19.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:18.807+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:19.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:19.054+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:19.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:19.054+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:19.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:19.054+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:19.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:19.055+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:19.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:19.317+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:19.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:19.317+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:19.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:19.317+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:19.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:19.318+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:20.062 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:19.779+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:20.062 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:19.779+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:20.062 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:19.779+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:20.062 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:19.780+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:20.341 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:20.062+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:20.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:20.062+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:20.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:20.062+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:20.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:20.063+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:20.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:16:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:16:20.398+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21822 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:16:20.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:20.342+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:20.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:20.342+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:20.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:20.342+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:20.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:20.343+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:21.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:20.750+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:21.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:20.750+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:21.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:20.750+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:21.089 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:20.750+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:21.384 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:21.088+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:21.384 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:21.088+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:21.384 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:21.088+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:21.384 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:21.089+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:21.717 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:21.384+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:21.717 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:21.384+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:21.717 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:21.384+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:21.717 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:21.385+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:21.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:21.753+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:21.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:21.753+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:21.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:21.753+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:21.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:21.754+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:22.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:22.067+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:22.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:22.067+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:22.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:22.067+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:22.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:22.068+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:22.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:22.432+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:22.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:22.432+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:22.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:22.432+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:22.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:22.433+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:22.735+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:22.735+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:22.735+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:23.088 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:22.736+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:23.461 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:23.088+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:23.461 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:23.088+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:23.462 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:23.088+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:23.462 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:23.089+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:23.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:23.709+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:23.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:23.709+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:23.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:23.709+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:23.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:23.710+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:23.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:23.462+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:23.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:23.462+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:23.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:23.462+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:23.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:23.462+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:24.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:24.084+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:24.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:24.084+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:24.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:24.084+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:24.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:24.084+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:24.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:24.729+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:24.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:24.729+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:24.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:24.729+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:24.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:24.729+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:24.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:24.501+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:24.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:24.501+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:24.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:24.501+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:24.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:24.501+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:25.401 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:25.049+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:25.402 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:25.049+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:25.402 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:25.049+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:25.402 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:25.049+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:25.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:16:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:16:25.400+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21908 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:16:25.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:25.527+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:25.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:25.527+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:25.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:25.527+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:25.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:25.528+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:26.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:25.768+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:26.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:25.768+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:26.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:25.768+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:26.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:25.769+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:26.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:26.009+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:26.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:26.009+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:26.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:26.009+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:26.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:26.010+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:26.816 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:26.554+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:26.817 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:26.554+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:26.817 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:26.554+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:26.817 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:26.555+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:27.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:26.966+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:27.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:26.966+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:27.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:26.966+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:27.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:26.967+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:27.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:26.817+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:27.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:26.817+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:27.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:26.817+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:27.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:26.817+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:27.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:27.798+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:27.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:27.798+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:27.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:27.798+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:27.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:27.798+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:27.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:27.590+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:27.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:27.590+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:27.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:27.590+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:27.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:27.591+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:28.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:27.993+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:28.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:27.993+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:28.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:27.993+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:28.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:27.994+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:28.964 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:28.602+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:28.964 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:28.602+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:28.965 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:28.602+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:28.965 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:28.602+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:28.965 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:28.751+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:28.965 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:28.751+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:28.965 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:28.751+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:28.965 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:28.752+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:29.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:28.964+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:29.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:28.964+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:29.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:28.964+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:29.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:28.965+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:29.915 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:29.745+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:29.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:29.745+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:29.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:29.745+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:29.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:29.745+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:29.916 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:29.602+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:29.916 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:29.602+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:29.916 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:29.602+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:29.917 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:29.603+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:30.182 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:29.916+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:30.183 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:29.916+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:30.183 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:29.916+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:30.183 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:29.916+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:30.473 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:16:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:16:30.402+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21991 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:16:30.921 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:30.743+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:30.921 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:30.743+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:30.922 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:30.743+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:30.922 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:30.744+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:30.922 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:30.646+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:30.922 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:30.646+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:30.922 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:30.646+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:30.922 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:30.647+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:31.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:30.922+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:31.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:30.922+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:31.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:30.922+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:31.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:30.922+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:31.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:31.733+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:31.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:31.733+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:31.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:31.733+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:31.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:31.734+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:31.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:31.652+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:31.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:31.652+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:31.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:31.652+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:31.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:31.652+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:32.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:31.951+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:32.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:31.951+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:32.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:31.951+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:32.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:31.952+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:32.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:32.610+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:32.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:32.610+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:32.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:32.610+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:32.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:32.611+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:32.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:32.775+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:32.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:32.775+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:32.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:32.775+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:32.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:32.776+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:33.414 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:32.977+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:33.415 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:32.977+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:33.415 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:32.977+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:33.415 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:32.978+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:33.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:33.577+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:33.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:33.577+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:33.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:33.577+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:33.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:33.578+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:34.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:33.971+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:34.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:33.971+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:34.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:33.971+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:34.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:33.972+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:34.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:33.736+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:34.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:33.736+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:34.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:33.736+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:34.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:33.737+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:34.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:34.715+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:34.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:34.715+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:34.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:34.715+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:34.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:34.716+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:34.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:34.627+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:34.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:34.627+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:34.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:34.627+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:34.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:34.628+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:35.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:34.952+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:35.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:34.952+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:35.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:34.952+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:35.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:34.953+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:35.681 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:16:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:16:35.405+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22077 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:16:35.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:35.633+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:35.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:35.633+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:35.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:35.633+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:35.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:35.633+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:35.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:35.913+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:35.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:35.913+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:35.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:35.913+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:35.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:35.914+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:35.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:35.682+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:35.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:35.682+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:35.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:35.682+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:35.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:35.682+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:36.956 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:36.618+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:36.956 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:36.618+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:36.956 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:36.618+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:36.956 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:36.619+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:36.957 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:36.651+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:36.957 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:36.651+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:36.957 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:36.651+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:36.957 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:36.651+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:37.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:36.956+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:37.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:36.956+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:37.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:36.956+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:37.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:36.957+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:37.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:37.631+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:37.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:37.631+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:37.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:37.631+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:37.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:37.632+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:37.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:37.634+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:37.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:37.634+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:37.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:37.634+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:37.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:37.634+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:38.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:37.949+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:38.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:37.949+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:38.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:37.949+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:38.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:37.950+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:38.924 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:38.668+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:38.924 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:38.668+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:38.925 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:38.668+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:38.925 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:38.669+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:38.925 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:38.675+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:38.925 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:38.675+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:38.925 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:38.675+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:38.926 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:38.675+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:39.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:38.924+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:39.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:38.924+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:39.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:38.925+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:39.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:38.925+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:39.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:39.891+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:39.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:39.891+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:39.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:39.891+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:39.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:39.891+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:39.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:39.679+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:39.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:39.679+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:39.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:39.679+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:39.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:39.680+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:39.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:39.685+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:39.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:39.685+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:39.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:39.685+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:39.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:39.686+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:40.473 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:16:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:16:40.407+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22160 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:16:40.936 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:40.678+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:40.936 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:40.678+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:40.936 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:40.678+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:40.936 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:40.679+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:40.937 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:40.690+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:40.937 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:40.690+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:40.937 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:40.690+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:40.937 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:40.691+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:41.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:40.936+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:41.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:40.936+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:41.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:40.936+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:41.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:40.937+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:41.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:41.657+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:41.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:41.657+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:41.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:41.657+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:41.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:41.658+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:41.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:41.890+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:41.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:41.890+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:41.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:41.890+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:41.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:41.891+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:41.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:41.673+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:41.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:41.674+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:41.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:41.674+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:41.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:41.674+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:42.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:42.637+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:42.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:42.637+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:42.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:42.637+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:42.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:42.638+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:42.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:42.692+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:42.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:42.692+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:42.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:42.692+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:42.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:42.693+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:42.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:42.843+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:42.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:42.843+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:42.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:42.843+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:42.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:42.844+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:43.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:43.795+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:43.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:43.795+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:43.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:43.795+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:43.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:43.795+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:43.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:43.712+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:43.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:43.712+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:43.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:43.712+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:43.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:43.713+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:43.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:43.648+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:43.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:43.648+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:43.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:43.648+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:43.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:43.649+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:44.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:44.658+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:44.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:44.658+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:44.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:44.658+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:44.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:44.659+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:44.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:44.814+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:44.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:44.814+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:44.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:44.814+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:44.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:44.815+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:44.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:44.739+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:44.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:44.739+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:44.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:44.739+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:44.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:44.739+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:45.679 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:16:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:16:45.409+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22246 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:16:45.973 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:45.836+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:45.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:45.836+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:45.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:45.836+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:45.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:45.837+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:45.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:45.679+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:45.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:45.679+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:45.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:45.679+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:45.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:45.680+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:45.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:45.695+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:45.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:45.695+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:45.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:45.695+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:45.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:45.696+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:46.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:46.676+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:46.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:46.676+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:46.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:46.676+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:46.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:46.676+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:46.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:46.883+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:46.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:46.883+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:46.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:46.883+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:46.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:46.884+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:46.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:46.683+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:46.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:46.683+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:46.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:46.683+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:46.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:46.684+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:47.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:47.672+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:47.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:47.672+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:47.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:47.672+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:47.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:47.673+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:47.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:47.662+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:47.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:47.662+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:47.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:47.662+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:47.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:47.663+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:47.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:47.874+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:47.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:47.874+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:47.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:47.874+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:47.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:47.874+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:48.891 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:48.636+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:48.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:48.636+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:48.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:48.636+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:48.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:48.638+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:48.893 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:48.649+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:48.893 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:48.649+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:48.893 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:48.649+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:48.893 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:48.649+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:49.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:48.891+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:49.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:48.891+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:49.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:48.891+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:49.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:48.892+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:49.852 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:49.687+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:49.852 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:49.687+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:49.853 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:49.687+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:49.853 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:49.688+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:49.853 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:49.600+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:49.853 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:49.600+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:49.854 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:49.600+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:49.854 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:49.601+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:50.183 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:49.852+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:50.184 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:49.852+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:50.184 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:49.852+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:50.184 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:49.853+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:50.473 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:16:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:16:50.412+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22329 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:16:50.899 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:50.650+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:50.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:50.650+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:50.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:50.650+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:50.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:50.651+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:50.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:50.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:50.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:50.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:50.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:50.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:50.901 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:50.627+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:51.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:50.898+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:51.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:50.898+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:51.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:50.898+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:51.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:50.899+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:51.889 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:51.619+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:51.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:51.619+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:51.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:51.619+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:51.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:51.620+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:51.890 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:51.655+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:51.891 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:51.655+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:51.891 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:51.655+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:51.891 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:51.656+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:52.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:51.889+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:52.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:51.889+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:52.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:51.889+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:52.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:51.890+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:52.882 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:52.584+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:52.882 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:52.584+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:52.883 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:52.584+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:52.883 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:52.585+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:52.883 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:52.703+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:52.883 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:52.703+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:52.883 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:52.703+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:52.883 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:52.704+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:53.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:52.882+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:53.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:52.882+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:53.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:52.882+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:53.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:52.883+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:53.851 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:53.753+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:53.851 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:53.753+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:53.852 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:53.753+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:53.852 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:53.754+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:53.852 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:53.542+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:53.852 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:53.542+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:53.852 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:53.542+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:53.852 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:53.543+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:54.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:53.852+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:54.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:53.852+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:54.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:53.852+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:54.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:53.853+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:54.878 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:54.803+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:54.878 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:54.803+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:54.878 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:54.803+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:54.879 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:54.803+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:54.879 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:54.580+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:54.879 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:54.580+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:54.879 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:54.580+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:54.879 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:54.581+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:55.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:54.878+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:55.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:54.878+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:55.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:54.878+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:55.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:54.879+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:55.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:16:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:16:55.414+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22415 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:16:55.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:55.545+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:55.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:55.545+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:55.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:55.545+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:55.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:55.546+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:56.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:55.769+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:56.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:55.769+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:56.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:55.769+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:56.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:55.769+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:56.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:55.929+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:56.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:55.929+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:56.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:55.929+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:56.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:55.930+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:56.955 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:56.778+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:56.955 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:56.778+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:56.956 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:56.778+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:56.956 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:56.779+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:56.956 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:56.580+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:56.956 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:56.580+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:56.956 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:56.580+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:56.956 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:56.580+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:57.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:56.955+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:57.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:56.955+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:57.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:56.955+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:57.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:56.956+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:57.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:57.735+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:57.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:57.735+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:57.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:57.735+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:57.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:57.736+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:57.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:57.624+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:57.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:57.624+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:57.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:57.624+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:57.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:57.625+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:58.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:57.978+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:58.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:57.978+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:58.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:57.978+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:58.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:57.979+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:58.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:58.625+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:58.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:58.625+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:58.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:58.625+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:58.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:58.626+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:58.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:58.761+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:58.953 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:58.761+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:58.953 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:58.761+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:58.953 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:58.762+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:59.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:58.951+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:59.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:58.951+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:59.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:58.951+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:16:59.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:58.952+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:16:59.955 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:59.730+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:59.955 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:59.730+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:59.956 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:59.730+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:16:59.956 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:16:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:16:59.731+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:16:59.956 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:59.585+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:59.956 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:59.585+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:59.956 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:59.585+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:16:59.957 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:16:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:16:59.586+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:00.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:59.955+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:00.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:59.955+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:00.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:59.955+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:00.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:16:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:16:59.956+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:00.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:17:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:17:00.416+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22498 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:17:00.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:00.539+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:00.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:00.539+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:00.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:00.539+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:00.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:00.540+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:01.062 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:00.748+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:01.062 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:00.748+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:01.062 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:00.748+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:01.062 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:00.749+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:01.063 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:00.918+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:01.063 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:00.918+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:01.063 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:00.918+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:01.063 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:00.918+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:01.928 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:01.570+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:01.928 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:01.570+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:01.929 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:01.570+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:01.929 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:01.570+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:01.929 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:01.758+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:01.929 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:01.758+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:01.929 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:01.758+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:01.929 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:01.759+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:02.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:01.929+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:02.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:01.929+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:02.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:01.929+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:02.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:01.929+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:02.956 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:02.770+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:02.957 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:02.770+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:02.957 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:02.770+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:02.957 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:02.771+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:02.957 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:02.582+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:02.957 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:02.582+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:02.958 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:02.582+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:02.958 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:02.582+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:03.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:02.957+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:03.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:02.957+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:03.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:02.957+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:03.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:02.958+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:03.922 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:03.750+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:03.922 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:03.750+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:03.922 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:03.750+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:03.923 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:03.751+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:03.923 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:03.610+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:03.923 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:03.610+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:03.923 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:03.610+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:03.923 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:03.610+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:04.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:03.922+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:04.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:03.922+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:04.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:03.922+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:04.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:03.923+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:04.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:04.561+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:04.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:04.561+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:04.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:04.561+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:04.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:04.562+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:04.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:04.789+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:04.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:04.789+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:04.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:04.789+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:04.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:04.790+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:05.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:04.971+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:05.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:04.971+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:05.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:04.971+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:05.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:04.972+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:05.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:17:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:17:05.418+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22584 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:17:05.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:05.514+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:05.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:05.514+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:05.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:05.514+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:05.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:05.515+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:06.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:05.987+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:06.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:05.987+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:06.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:05.987+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:06.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:05.988+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:06.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:05.817+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:06.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:05.817+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:06.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:05.817+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:06.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:05.817+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:06.792 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:06.483+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:06.792 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:06.483+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:06.792 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:06.483+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:06.792 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:06.484+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:07.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:06.792+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:07.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:06.792+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:07.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:06.792+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:07.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:06.793+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:07.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:06.998+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:07.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:06.998+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:07.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:06.998+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:07.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:06.999+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:07.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:07.438+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:07.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:07.438+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:07.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:07.438+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:07.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:07.439+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:08.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:07.794+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:08.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:07.794+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:08.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:07.794+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:08.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:07.794+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:08.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:08.002+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:08.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:08.002+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:08.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:08.002+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:08.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:08.003+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:08.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:08.396+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:08.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:08.396+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:08.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:08.396+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:08.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:08.396+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:09.028 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:08.749+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:09.029 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:08.749+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:09.029 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:08.749+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:09.029 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:08.750+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:09.416 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:09.029+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:09.416 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:09.029+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:09.416 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:09.029+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:09.416 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:09.030+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:09.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:09.416+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:09.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:09.416+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:09.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:09.416+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:09.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:09.417+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:10.056 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:09.784+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:10.057 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:09.784+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:10.057 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:09.784+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:10.057 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:09.785+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:10.370 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:10.057+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:10.370 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:10.057+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:10.370 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:10.057+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:10.370 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:10.057+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:10.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:17:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:17:10.421+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22667 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:17:10.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:10.370+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:10.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:10.370+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:10.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:10.370+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:10.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:10.371+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:11.098 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:10.816+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:11.098 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:10.816+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:11.098 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:10.816+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:11.099 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:10.817+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:11.368 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:11.098+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:11.368 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:11.098+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:11.368 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:11.098+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:11.369 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:11.099+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:11.721 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:11.369+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:11.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:11.369+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:11.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:11.369+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:11.722 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:11.369+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:11.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:11.788+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:11.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:11.788+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:11.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:11.788+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:11.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:11.789+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:12.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:12.323+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:12.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:12.323+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:12.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:12.323+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:12.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:12.323+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:12.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:12.108+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:12.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:12.108+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:12.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:12.108+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:12.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:12.108+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:13.139 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:12.758+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:13.139 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:12.758+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:13.140 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:12.758+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:13.140 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:12.758+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:13.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:13.359+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:13.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:13.359+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:13.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:13.359+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:13.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:13.360+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:13.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:13.139+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:13.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:13.139+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:13.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:13.139+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:13.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:13.140+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:13.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:13.732+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:13.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:13.732+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:13.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:13.732+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:13.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:13.733+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:14.400 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:14.115+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:14.401 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:14.115+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:14.401 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:14.115+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:14.401 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:14.116+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:14.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:14.401+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:14.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:14.401+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:14.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:14.401+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:14.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:14.401+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:15.107 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:14.752+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:15.107 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:14.752+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:15.107 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:14.752+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:15.107 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:14.753+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:15.423 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:15.107+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:15.423 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:15.107+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:15.423 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:15.107+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:15.424 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:15.108+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:15.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:17:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:17:15.423+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22753 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:17:15.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:15.436+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:15.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:15.436+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:15.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:15.436+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:15.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:15.437+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:16.108 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:15.779+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:16.109 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:15.779+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:16.109 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:15.779+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:16.109 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:15.780+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:16.367 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:16.109+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:16.368 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:16.109+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:16.368 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:16.109+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:16.368 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:16.109+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:16.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:16.427+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:16.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:16.427+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:16.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:16.427+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:16.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:16.428+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:17.062 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:16.748+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:17.062 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:16.748+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:17.062 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:16.748+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:17.062 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:16.748+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:17.425 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:17.062+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:17.425 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:17.062+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:17.425 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:17.062+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:17.425 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:17.063+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:17.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:17.425+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:17.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:17.425+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:17.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:17.425+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:17.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:17.426+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:18.030 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:17.764+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:18.030 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:17.764+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:18.030 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:17.764+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:18.031 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:17.765+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:18.463 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:18.030+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:18.464 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:18.030+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:18.464 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:18.030+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:18.464 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:18.031+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:18.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:18.463+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:18.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:18.463+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:18.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:18.463+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:18.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:18.464+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:19.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:19.011+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:19.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:19.011+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:19.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:19.011+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:19.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:19.012+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:19.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:18.764+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:19.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:18.764+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:19.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:18.764+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:19.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:18.765+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:19.717 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:19.474+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:19.717 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:19.474+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:19.718 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:19.474+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:19.718 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:19.475+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:19.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:19.717+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:19.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:19.717+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:19.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:19.717+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:19.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:19.718+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:20.425 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:20.045+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:20.426 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:20.045+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:20.426 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:20.045+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:20.426 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:20.046+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:20.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:17:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:17:20.425+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22836 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:17:20.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:20.501+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:20.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:20.501+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:20.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:20.501+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:20.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:20.502+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:21.024 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:20.731+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:21.024 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:20.731+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:21.025 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:20.731+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:21.025 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:20.732+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:21.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:21.023+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:21.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:21.023+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:21.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:21.023+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:21.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:21.024+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:21.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:21.732+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:21.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:21.732+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:21.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:21.732+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:21.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:21.733+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:21.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:21.529+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:21.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:21.529+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:21.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:21.529+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:21.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:21.530+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:22.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:22.073+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:22.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:22.073+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:22.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:22.073+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:22.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:22.073+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:22.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:22.692+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:22.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:22.692+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:22.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:22.692+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:22.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:22.693+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:22.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:22.530+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:22.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:22.530+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:22.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:22.530+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:22.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:22.531+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:23.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:23.063+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:23.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:23.063+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:23.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:23.063+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:23.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:23.064+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:23.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:23.666+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:23.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:23.666+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:23.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:23.666+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:23.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:23.667+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:23.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:23.574+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:23.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:23.574+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:23.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:23.574+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:23.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:23.575+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:24.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:24.062+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:24.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:24.062+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:24.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:24.062+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:24.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:24.062+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:24.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:24.567+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:24.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:24.567+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:24.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:24.567+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:24.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:24.567+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:24.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:24.637+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:24.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:24.637+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:24.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:24.637+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:24.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:24.638+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:25.428 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:25.100+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:25.428 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:25.100+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:25.428 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:25.100+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:25.428 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:25.101+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:25.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:17:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:17:25.427+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22922 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:17:25.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:25.549+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:25.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:25.549+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:25.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:25.549+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:25.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:25.550+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:25.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:25.632+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:25.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:25.632+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:25.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:25.632+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:25.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:25.632+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:26.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:26.099+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:26.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:26.099+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:26.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:26.099+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:26.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:26.099+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:26.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:26.543+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:26.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:26.543+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:26.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:26.543+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:26.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:26.543+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:26.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:26.646+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:26.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:26.646+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:26.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:26.646+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:26.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:26.647+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:27.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:27.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:27.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:27.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:27.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:27.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:27.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:27.054+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:27.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:27.581+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:27.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:27.581+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:27.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:27.581+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:27.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:27.582+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:27.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:27.615+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:27.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:27.615+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:27.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:27.615+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:27.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:27.615+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:28.462 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:28.090+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:28.462 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:28.090+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:28.462 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:28.090+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:28.462 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:28.091+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:28.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:28.582+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:28.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:28.582+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:28.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:28.582+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:28.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:28.583+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:28.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:28.554+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:28.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:28.554+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:28.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:28.554+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:28.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:28.554+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:29.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:29.096+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:29.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:29.096+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:29.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:29.096+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:29.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:29.097+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:29.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:29.622+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:29.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:29.622+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:29.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:29.622+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:29.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:29.623+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:29.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:29.509+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:29.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:29.509+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:29.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:29.509+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:29.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:29.509+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:30.430 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:30.113+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:30.431 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:30.113+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:30.431 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:30.113+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:30.431 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:30.113+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:30.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:17:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:17:30.429+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23005 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:17:30.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:30.668+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:30.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:30.668+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:30.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:30.668+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:30.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:30.669+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:30.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:30.534+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:30.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:30.534+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:30.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:30.534+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:30.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:30.534+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:31.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:31.133+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:31.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:31.133+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:31.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:31.133+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:31.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:31.133+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:31.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:31.685+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:31.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:31.685+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:31.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:31.685+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:31.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:31.686+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:31.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:31.536+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:31.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:31.536+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:31.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:31.536+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:31.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:31.537+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:32.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:32.117+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:32.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:32.117+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:32.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:32.117+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:32.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:32.118+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:32.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:32.701+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:32.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:32.701+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:32.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:32.701+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:32.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:32.703+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:32.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:32.582+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:32.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:32.582+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:32.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:32.582+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:32.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:32.583+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:33.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:33.146+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:33.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:33.146+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:33.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:33.146+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:33.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:33.147+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:33.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:33.665+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:33.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:33.665+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:33.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:33.665+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:33.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:33.666+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:33.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:33.615+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:33.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:33.615+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:33.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:33.615+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:33.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:33.616+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:34.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:34.126+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:34.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:34.126+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:34.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:34.126+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:34.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:34.127+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:34.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:34.648+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:34.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:34.648+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:34.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:34.648+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:34.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:34.649+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:34.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:34.603+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:34.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:34.603+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:34.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:34.603+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:34.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:34.604+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:35.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:35.109+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:35.433 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:35.109+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:35.433 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:35.109+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:35.433 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:35.110+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:35.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:17:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:17:35.432+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23091 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:17:35.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:35.638+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:35.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:35.638+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:35.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:35.638+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:35.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:35.639+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:35.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:35.627+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:35.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:35.627+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:35.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:35.627+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:35.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:35.627+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:36.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:36.075+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:36.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:36.075+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:36.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:36.075+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:36.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:36.075+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:36.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:36.589+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:36.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:36.589+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:36.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:36.589+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:36.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:36.590+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:36.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:36.610+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:36.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:36.610+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:36.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:36.610+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:36.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:36.611+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:37.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:37.077+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:37.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:37.077+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:37.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:37.077+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:37.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:37.078+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:37.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:37.540+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:37.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:37.540+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:37.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:37.540+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:37.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:37.541+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:37.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:37.563+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:37.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:37.563+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:37.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:37.563+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:37.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:37.564+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:38.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:38.101+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:38.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:38.101+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:38.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:38.101+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:38.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:38.102+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:38.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:38.600+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:38.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:38.600+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:38.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:38.600+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:38.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:38.600+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:38.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:38.508+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:38.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:38.508+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:38.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:38.508+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:38.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:38.509+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:39.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:39.067+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:39.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:39.067+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:39.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:39.067+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:39.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:39.067+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:39.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:39.539+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:39.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:39.539+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:39.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:39.539+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:39.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:39.540+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:39.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:39.552+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:39.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:39.552+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:39.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:39.552+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:39.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:39.552+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:40.435 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:40.032+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:40.435 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:40.032+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:40.435 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:40.032+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:40.435 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:40.033+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:40.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:17:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:17:40.435+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23174 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:17:40.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:40.574+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:40.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:40.574+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:40.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:40.574+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:40.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:40.574+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:40.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:40.530+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:40.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:40.530+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:40.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:40.530+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:40.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:40.531+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:41.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:40.997+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:41.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:40.997+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:41.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:40.997+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:41.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:40.998+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:41.962 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:41.493+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:41.962 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:41.493+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:41.962 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:41.493+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:41.963 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:41.494+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:41.963 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:41.574+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:41.963 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:41.574+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:41.963 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:41.574+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:41.963 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:41.574+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:42.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:41.962+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:42.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:41.962+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:42.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:41.962+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:42.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:41.963+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:42.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:42.467+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:42.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:42.467+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:42.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:42.467+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:42.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:42.468+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:42.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:42.570+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:42.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:42.570+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:42.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:42.570+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:42.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:42.570+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:43.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:42.986+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:43.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:42.986+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:43.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:42.986+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:43.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:42.987+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:43.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:43.438+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:43.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:43.438+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:43.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:43.438+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:43.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:43.439+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:43.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:43.561+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:43.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:43.561+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:43.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:43.561+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:43.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:43.562+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:44.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:44.016+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:44.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:44.016+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:44.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:44.016+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:44.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:44.016+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:44.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:44.524+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:44.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:44.524+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:44.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:44.524+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:44.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:44.524+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:44.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:44.489+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:44.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:44.489+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:44.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:44.489+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:44.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:44.489+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:45.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:45.044+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:45.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:45.044+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:45.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:45.044+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:45.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:45.045+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:45.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:17:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:17:45.438+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23260 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:17:45.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:45.513+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:45.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:45.513+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:45.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:45.513+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:45.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:45.514+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:45.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:45.500+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:45.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:45.500+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:45.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:45.500+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:45.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:45.501+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:46.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:46.050+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:46.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:46.050+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:46.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:46.050+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:46.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:46.051+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:46.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:46.481+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:46.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:46.481+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:46.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:46.481+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:46.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:46.482+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:46.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:46.546+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:46.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:46.546+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:46.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:46.546+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:46.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:46.547+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:47.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:47.098+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:47.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:47.098+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:47.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:47.098+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:47.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:47.099+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:47.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:47.445+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:47.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:47.445+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:47.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:47.445+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:47.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:47.446+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:47.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:47.569+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:47.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:47.569+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:47.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:47.569+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:47.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:47.569+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:48.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:48.117+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:48.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:48.117+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:48.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:48.117+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:48.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:48.118+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:48.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:48.595+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:48.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:48.595+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:48.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:48.595+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:48.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:48.596+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:48.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:48.437+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:48.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:48.437+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:48.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:48.437+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:48.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:48.438+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:49.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:49.150+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:49.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:49.150+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:49.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:49.150+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:49.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:49.150+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:49.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:49.438+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:49.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:49.438+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:49.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:49.438+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:49.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:49.439+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:49.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:49.644+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:49.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:49.644+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:49.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:49.644+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:49.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:49.645+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:50.429 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:50.122+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:50.429 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:50.122+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:50.429 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:50.122+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:50.429 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:50.123+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:50.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:17:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:17:50.440+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23343 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:17:50.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:50.429+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:50.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:50.429+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:50.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:50.429+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:50.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:50.430+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:50.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:50.604+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:50.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:50.604+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:50.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:50.604+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:50.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:50.604+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:51.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:51.115+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:51.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:51.115+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:51.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:51.115+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:51.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:51.116+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:51.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:51.583+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:51.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:51.583+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:51.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:51.583+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:51.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:51.584+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:51.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:51.446+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:51.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:51.446+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:51.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:51.446+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:51.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:51.447+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:52.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:52.109+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:52.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:52.109+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:52.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:52.109+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:52.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:52.109+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:52.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:52.474+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:52.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:52.474+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:52.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:52.474+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:52.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:52.475+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:52.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:52.597+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:52.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:52.597+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:52.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:52.597+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:52.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:52.598+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:53.440 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:53.105+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:53.440 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:53.105+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:53.441 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:53.105+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:53.441 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:53.106+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:53.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:53.440+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:53.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:53.440+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:53.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:53.440+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:53.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:53.441+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:53.728 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:53.572+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:53.728 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:53.572+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:53.728 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:53.572+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:53.729 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:53.572+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:54.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:54.126+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:54.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:54.126+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:54.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:54.126+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:54.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:54.127+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:54.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:54.438+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:54.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:54.438+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:54.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:54.438+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:54.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:54.439+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:54.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:54.607+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:54.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:54.607+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:54.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:54.607+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:54.607+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:55.429 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:55.126+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:55.429 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:55.126+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:55.429 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:55.126+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:55.429 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:55.127+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:55.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:17:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:17:55.442+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23429 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:17:55.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:55.573+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:55.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:55.573+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:55.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:55.573+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:55.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:55.574+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:55.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:55.429+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:55.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:55.429+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:55.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:55.429+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:55.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:55.430+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:56.359 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:56.082+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:56.359 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:56.082+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:56.359 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:56.082+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:56.359 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:56.083+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:56.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:56.427+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:56.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:56.427+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:56.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:56.427+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:56.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:56.428+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:56.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:56.590+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:56.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:56.590+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:56.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:56.590+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:56.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:56.590+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:57.440 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:57.042+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:57.441 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:57.042+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:57.441 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:57.042+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:57.441 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:57.043+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:57.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:57.441+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:57.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:57.441+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:57.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:57.441+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:57.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:57.442+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:57.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:57.548+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:57.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:57.548+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:57.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:57.549+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:57.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:57.549+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:58.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:58.077+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:58.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:58.077+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:58.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:58.077+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:58.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:58.078+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:58.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:58.480+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:58.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:58.480+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:58.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:58.480+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:58.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:58.480+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:58.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:58.589+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:58.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:58.589+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:58.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:58.589+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:58.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:58.590+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:59.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:59.029+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:59.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:59.029+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:59.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:59.029+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:17:59.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:17:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:17:59.030+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:17:59.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:59.508+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:59.508+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:59.508+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:17:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:17:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:17:59.509+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:17:59.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:59.601+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:59.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:59.601+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:59.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:59.601+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:17:59.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:17:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:17:59.601+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:00.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:00.079+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:00.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:00.079+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:00.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:00.079+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:00.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:00.080+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:00.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:18:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:18:00.445+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23512 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:18:00.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:00.483+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:00.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:00.483+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:00.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:00.483+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:00.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:00.484+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:00.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:00.638+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:00.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:00.638+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:00.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:00.638+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:00.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:00.639+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:01.464 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:01.104+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:01.465 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:01.104+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:01.465 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:01.104+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:01.465 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:01.105+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:01.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:01.594+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:01.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:01.594+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:01.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:01.594+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:01.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:01.595+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:01.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:01.465+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:01.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:01.465+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:01.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:01.465+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:01.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:01.465+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:02.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:02.100+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:02.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:02.100+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:02.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:02.100+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:02.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:02.101+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:02.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:02.490+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:02.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:02.490+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:02.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:02.490+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:02.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:02.490+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:02.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:02.588+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:02.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:02.588+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:02.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:02.588+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:02.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:02.589+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:03.463 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:03.056+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:03.464 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:03.056+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:03.464 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:03.056+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:03.464 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:03.057+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:03.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:03.464+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:03.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:03.464+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:03.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:03.464+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:03.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:03.465+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:03.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:03.561+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:03.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:03.561+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:03.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:03.561+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:03.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:03.561+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:04.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:04.085+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:04.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:04.085+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:04.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:04.085+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:04.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:04.086+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:04.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:04.592+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:04.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:04.592+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:04.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:04.592+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:04.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:04.593+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:04.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:04.445+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:04.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:04.445+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:04.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:04.445+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:04.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:04.446+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:05.398 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:05.036+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:05.399 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:05.036+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:05.399 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:05.036+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:05.399 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:05.037+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:05.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:18:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:18:05.447+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23606 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:18:05.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:05.548+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:05.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:05.548+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:05.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:05.548+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:05.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:05.548+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:05.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:05.399+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:05.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:05.399+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:05.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:05.399+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:05.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:05.400+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:06.410 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:06.071+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:06.410 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:06.071+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:06.410 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:06.071+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:06.410 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:06.072+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:06.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:06.410+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:06.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:06.410+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:06.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:06.410+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:06.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:06.411+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:06.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:06.582+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:06.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:06.582+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:06.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:06.582+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:06.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:06.582+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:07.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:07.044+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:07.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:07.044+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:07.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:07.044+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:07.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:07.045+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:07.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:07.437+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:07.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:07.437+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:07.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:07.437+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:07.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:07.438+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:07.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:07.543+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:07.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:07.543+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:07.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:07.543+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:07.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:07.544+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:08.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:08.009+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:08.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:08.009+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:08.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:08.009+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:08.467 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:08.010+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:08.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:08.501+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:08.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:08.501+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:08.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:08.501+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:08.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:08.501+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:08.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:08.467+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:08.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:08.467+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:08.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:08.467+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:08.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:08.468+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:09.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:09.056+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:09.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:09.056+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:09.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:09.056+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:09.421 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:09.057+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:09.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:09.515+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:09.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:09.515+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:09.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:09.515+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:09.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:09.516+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:09.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:09.420+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:09.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:09.420+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:09.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:09.420+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:09.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:09.421+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:10.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:10.020+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:10.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:10.020+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:10.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:10.020+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:10.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:10.021+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:10.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:18:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:18:10.450+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23689 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:18:10.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:10.444+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:10.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:10.444+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:10.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:10.444+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:10.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:10.445+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:10.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:10.487+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:10.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:10.487+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:10.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:10.487+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:10.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:10.488+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:11.399 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:11.020+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:11.399 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:11.020+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:11.399 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:11.020+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:11.399 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:11.021+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:11.717 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:11.399+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:11.718 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:11.399+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:11.718 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:11.399+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:11.718 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:11.400+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:11.718 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:11.458+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:11.718 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:11.458+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:11.719 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:11.458+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:11.719 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:11.459+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:12.376 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:11.988+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:12.377 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:11.988+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:12.377 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:11.988+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:12.377 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:11.989+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:12.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:12.376+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:12.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:12.376+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:12.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:12.376+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:12.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:12.377+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:12.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:12.414+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:12.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:12.414+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:12.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:12.414+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:12.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:12.414+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:13.373 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:13.014+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:13.373 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:13.014+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:13.374 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:13.014+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:13.374 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:13.015+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:13.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:13.400+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:13.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:13.400+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:13.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:13.400+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:13.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:13.401+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:13.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:13.373+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:13.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:13.373+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:13.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:13.373+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:13.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:13.374+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:14.331 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:13.998+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:14.332 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:13.998+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:14.332 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:13.998+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:14.332 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:13.999+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:14.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:14.331+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:14.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:14.331+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:14.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:14.331+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:14.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:14.332+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:14.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:14.438+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:14.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:14.438+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:14.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:14.438+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:14.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:14.439+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:15.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:15.016+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:15.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:15.016+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:15.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:15.016+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:15.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:15.017+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:15.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:18:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:18:15.452+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23774 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:18:15.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:15.408+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:15.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:15.409+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:15.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:15.409+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:15.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:15.409+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:15.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:15.330+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:15.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:15.330+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:15.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:15.330+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:15.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:15.331+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:16.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:15.971+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:16.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:15.972+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:16.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:15.972+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:16.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:15.973+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:16.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:16.379+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:16.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:16.379+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:16.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:16.379+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:16.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:16.380+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:16.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:16.353+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:16.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:16.353+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:16.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:16.353+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:16.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:16.354+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:17.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:16.981+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:17.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:16.981+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:17.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:16.981+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:17.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:16.982+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:17.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:17.354+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:17.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:17.354+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:17.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:17.354+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:17.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:17.354+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:17.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:17.342+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:17.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:17.342+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:17.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:17.342+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:17.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:17.343+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:18.343 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:17.976+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:18.343 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:17.976+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:18.343 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:17.976+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:18.344 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:17.977+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:18.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:18.343+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:18.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:18.343+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:18.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:18.343+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:18.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:18.344+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:18.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:18.354+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:18.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:18.354+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:18.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:18.354+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:18.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:18.355+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:19.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:18.938+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:19.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:18.938+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:19.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:18.938+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:19.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:18.938+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:19.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:19.330+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:19.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:19.330+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:19.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:19.330+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:19.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:19.331+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:19.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:19.369+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:19.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:19.369+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:19.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:19.369+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:19.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:19.369+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:20.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:19.962+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:20.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:19.962+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:20.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:19.962+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:20.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:19.963+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:20.575 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:20.290+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:20.575 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:20.290+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:20.575 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:20.290+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:20.575 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:20.291+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:20.576 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:18:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:18:20.455+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23857 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:18:20.576 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:20.327+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:20.576 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:20.327+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:20.576 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:20.327+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:20.577 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:20.328+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:21.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:20.920+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:21.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:20.920+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:21.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:20.920+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:21.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:20.920+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:21.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:21.317+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:21.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:21.317+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:21.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:21.317+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:21.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:21.318+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:21.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:21.290+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:21.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:21.290+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:21.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:21.290+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:21.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:21.291+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:22.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:21.909+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:22.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:21.909+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:22.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:21.909+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:22.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:21.910+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:22.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:22.318+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:22.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:22.318+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:22.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:22.318+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:22.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:22.319+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:22.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:22.326+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:22.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:22.326+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:22.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:22.326+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:22.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:22.326+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:23.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:22.890+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:23.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:22.890+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:23.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:22.890+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:23.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:22.891+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:23.603 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:23.330+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:23.603 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:23.330+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:23.604 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:23.330+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:23.604 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:23.331+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:23.604 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:23.339+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:23.604 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:23.339+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:23.604 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:23.339+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:23.605 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:23.339+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:24.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:23.904+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:24.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:23.904+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:24.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:23.904+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:24.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:23.905+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:24.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:24.336+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:24.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:24.336+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:24.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:24.336+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:24.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:24.337+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:24.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:24.364+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:24.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:24.364+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:24.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:24.364+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:24.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:24.365+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:25.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:24.949+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:25.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:24.949+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:25.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:24.949+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:25.227 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:24.950+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:25.579 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:18:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:18:25.458+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23943 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:18:25.579 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:25.288+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:25.579 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:25.288+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:25.579 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:25.288+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:25.579 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:25.289+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:25.580 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:25.324+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:25.580 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:25.324+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:25.580 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:25.324+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:25.580 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:25.325+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:26.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:25.958+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:26.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:25.958+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:26.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:25.958+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:26.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:25.959+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:26.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:26.288+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:26.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:26.288+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:26.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:26.288+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:26.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:26.289+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:26.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:26.291+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:26.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:26.291+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:26.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:26.291+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:26.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:26.292+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:27.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:26.968+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:27.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:26.968+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:27.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:26.968+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:27.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:26.969+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:27.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:27.254+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:27.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:27.254+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:27.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:27.254+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:27.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:27.254+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:27.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:27.316+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:27.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:27.316+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:27.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:27.316+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:27.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:27.317+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:28.343 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:28.218+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:28.343 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:28.218+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:28.344 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:28.218+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:28.344 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:28.219+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:28.344 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:27.986+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:28.344 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:27.986+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:28.344 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:27.986+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:28.344 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:27.987+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:28.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:28.343+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:28.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:28.343+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:28.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:28.343+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:28.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:28.344+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:29.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:28.994+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:29.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:28.994+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:29.245 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:28.994+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:29.246 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:28.994+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:29.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:29.254+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:29.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:29.254+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:29.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:29.254+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:29.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:29.255+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:29.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:29.323+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:29.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:29.323+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:29.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:29.323+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:29.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:29.323+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:30.330 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:30.029+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:30.331 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:30.029+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:30.331 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:30.029+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:30.331 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:30.030+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:30.331 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:30.235+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:30.331 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:30.235+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:30.331 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:30.235+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:30.332 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:30.236+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:30.584 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:18:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:18:30.460+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24026 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:18:30.584 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:30.330+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:30.585 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:30.330+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:30.585 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:30.330+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:30.585 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:30.331+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:31.370 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:31.071+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:31.371 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:31.071+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:31.371 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:31.071+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:31.371 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:31.072+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:31.371 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:31.247+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:31.371 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:31.247+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:31.372 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:31.247+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:31.372 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:31.248+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:31.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:31.370+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:31.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:31.370+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:31.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:31.370+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:31.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:31.371+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:32.405 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:32.058+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:32.405 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:32.058+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:32.405 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:32.058+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:32.405 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:32.059+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:32.405 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:32.228+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:32.406 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:32.228+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:32.406 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:32.228+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:32.406 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:32.229+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:32.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:32.405+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:32.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:32.405+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:32.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:32.405+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:32.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:32.406+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:33.366 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:33.098+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:33.367 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:33.098+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:33.367 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:33.098+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:33.367 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:33.099+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:33.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:33.271+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:33.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:33.271+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:33.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:33.271+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:33.368 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:33.272+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:33.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:33.366+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:33.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:33.366+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:33.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:33.366+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:33.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:33.367+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:34.368 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:34.064+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:34.368 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:34.064+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:34.368 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:34.064+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:34.368 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:34.065+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:34.368 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:34.237+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:34.369 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:34.237+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:34.369 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:34.237+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:34.369 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:34.237+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:34.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:34.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:34.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:34.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:34.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:34.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:34.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:34.368+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:35.375 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:35.062+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:35.376 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:35.062+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:35.376 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:35.062+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:35.376 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:35.063+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:35.376 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:35.257+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:35.376 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:35.257+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:35.376 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:35.257+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:35.377 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:35.258+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:35.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:18:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:18:35.463+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24110 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:18:35.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:35.375+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:35.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:35.375+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:35.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:35.375+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:35.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:35.376+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:36.294 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:36.038+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:36.294 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:36.038+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:36.294 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:36.038+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:36.294 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:36.039+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:36.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:36.301+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:36.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:36.301+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:36.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:36.301+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:36.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:36.302+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:36.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:36.359+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:36.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:36.359+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:36.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:36.359+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:36.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:36.359+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:37.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:37.311+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:37.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:37.311+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:37.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:37.311+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:37.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:37.312+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:37.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:37.313+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:37.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:37.313+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:37.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:37.313+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:37.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:37.313+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:37.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:37.086+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:37.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:37.086+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:37.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:37.086+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:37.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:37.087+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:38.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:38.072+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:38.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:38.072+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:38.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:38.072+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:38.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:38.073+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:38.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:38.287+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:38.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:38.287+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:38.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:38.287+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:38.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:38.288+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:38.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:38.299+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:38.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:38.299+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:38.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:38.299+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:38.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:38.299+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:39.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:39.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:39.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:39.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:39.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:39.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:39.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:39.053+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:39.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:39.297+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:39.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:39.297+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:39.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:39.297+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:39.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:39.298+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:39.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:39.286+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:39.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:39.286+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:39.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:39.286+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:39.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:39.286+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:40.310 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:40.052+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:40.310 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:40.052+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:40.310 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:40.052+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:40.310 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:40.053+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:40.589 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:18:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:18:40.465+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24194 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:18:40.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:40.310+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:40.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:40.310+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:40.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:40.310+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:40.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:40.311+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:40.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:40.318+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:40.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:40.318+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:40.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:40.318+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:40.591 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:40.319+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:41.314 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:41.044+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:41.314 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:41.044+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:41.314 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:41.044+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:41.314 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:41.044+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:41.315 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:41.262+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:41.315 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:41.262+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:41.315 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:41.263+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:41.315 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:41.263+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:41.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:41.313+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:41.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:41.313+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:41.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:41.313+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:41.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:41.313+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:42.302 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:42.262+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:42.302 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:42.262+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:42.302 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:42.262+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:42.303 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:42.264+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:42.303 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:42.015+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:42.303 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:42.015+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:42.303 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:42.015+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:42.303 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:42.016+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:42.723 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:42.301+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:42.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:42.301+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:42.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:42.301+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:42.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:42.302+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:43.305 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:43.051+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:43.306 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:43.051+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:43.306 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:43.051+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:43.306 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:43.052+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:43.306 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:43.234+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:43.306 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:43.234+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:43.307 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:43.234+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:43.307 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:43.235+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:43.605 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:43.305+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:43.605 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:43.305+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:43.605 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:43.305+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:43.605 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:43.305+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:44.266 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:44.007+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:44.266 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:44.007+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:44.266 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:44.007+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:44.266 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:44.008+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:44.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:44.270+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:44.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:44.270+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:44.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:44.270+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:44.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:44.271+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:44.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:44.265+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:44.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:44.265+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:44.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:44.265+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:44.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:44.265+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:45.298 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:45.014+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:45.299 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:45.014+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:45.299 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:45.014+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:45.299 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:45.015+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:45.299 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:45.224+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:45.299 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:45.224+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:45.300 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:45.224+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:45.300 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:45.224+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:45.591 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:18:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:18:45.467+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24279 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:18:45.592 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:45.298+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:45.592 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:45.298+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:45.592 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:45.298+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:45.592 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:45.298+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:46.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:46.055+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:46.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:46.055+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:46.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:46.055+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:46.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:46.056+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:46.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:46.261+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:46.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:46.261+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:46.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:46.261+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:46.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:46.261+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:46.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:46.181+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:46.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:46.181+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:46.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:46.181+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:46.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:46.182+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:47.375 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:47.086+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:47.376 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:47.086+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:47.376 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:47.086+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:47.376 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:47.087+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:47.377 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:47.228+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:47.377 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:47.228+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:47.377 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:47.228+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:47.377 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:47.229+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:47.377 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:47.147+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:47.378 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:47.147+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:47.378 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:47.147+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:47.378 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:47.147+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:48.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:48.079+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:48.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:48.079+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:48.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:48.079+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:48.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:48.079+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:48.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:48.195+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:48.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:48.195+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:48.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:48.195+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:48.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:48.196+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:48.477 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:48.183+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:48.477 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:48.183+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:48.477 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:48.183+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:48.478 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:48.184+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:49.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:49.221+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:49.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:49.221+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:49.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:49.221+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:49.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:49.222+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:49.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:49.090+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:49.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:49.090+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:49.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:49.090+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:49.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:49.091+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:49.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:49.195+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:49.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:49.195+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:49.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:49.195+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:49.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:49.196+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:50.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:50.196+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:50.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:50.196+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:50.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:50.196+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:50.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:50.196+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:50.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:50.120+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:50.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:50.120+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:50.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:50.120+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:50.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:50.121+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:50.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:50.171+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:50.472 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:50.171+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:50.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:50.171+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:50.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:50.172+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:50.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:18:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:18:50.469+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24363 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:18:51.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:51.161+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:51.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:51.161+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:51.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:51.161+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:51.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:51.162+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:51.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:51.189+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:51.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:51.189+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:51.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:51.189+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:51.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:51.189+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:51.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:51.120+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:51.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:51.120+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:51.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:51.120+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:51.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:51.121+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:52.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:52.126+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:52.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:52.126+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:52.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:52.126+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:52.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:52.127+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:52.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:52.160+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:52.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:52.160+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:52.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:52.160+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:52.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:52.161+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:52.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:52.195+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:52.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:52.195+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:52.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:52.195+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:52.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:52.196+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:53.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:53.088+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:53.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:53.088+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:53.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:53.088+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:53.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:53.089+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:53.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:53.126+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:53.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:53.126+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:53.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:53.126+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:53.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:53.127+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:53.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:53.152+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:53.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:53.152+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:53.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:53.152+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:53.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:53.153+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:54.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:54.127+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:54.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:54.127+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:54.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:54.127+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:54.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:54.127+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:54.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:54.101+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:54.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:54.101+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:54.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:54.101+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:54.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:54.101+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:54.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:54.085+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:54.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:54.085+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:54.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:54.085+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:54.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:54.086+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:55.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:55.148+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:55.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:55.148+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:55.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:55.148+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:55.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:55.149+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:55.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:55.082+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:55.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:55.082+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:55.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:55.082+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:55.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:55.083+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:55.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:55.050+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:55.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:55.050+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:55.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:55.050+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:55.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:55.051+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:55.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:18:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:18:55.472+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24448 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:18:56.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:56.011+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:56.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:56.011+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:56.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:56.011+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:56.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:56.012+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:56.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:56.050+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:56.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:56.050+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:56.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:56.050+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:56.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:56.051+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:56.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:56.148+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:56.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:56.148+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:56.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:56.148+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:56.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:56.149+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:57.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:57.004+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:57.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:57.004+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:57.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:57.004+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:57.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:57.004+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:57.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:57.056+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:57.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:57.056+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:57.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:57.056+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:57.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:57.057+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:57.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:57.129+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:57.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:57.129+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:57.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:57.129+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:57.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:57.130+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:58.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:58.018+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:58.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:58.018+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:58.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:58.018+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:58.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:58.018+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:58.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:58.036+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:58.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:58.036+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:58.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:58.036+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:58.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:58.036+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:58.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:58.174+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:58.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:58.174+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:58.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:58.174+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:58.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:58.175+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:18:59.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:59.047+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:59.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:59.048+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:59.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:59.048+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:18:59.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:18:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:18:59.048+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:59.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:59.024+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:59.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:59.024+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:59.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:59.024+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:18:59.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:18:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:18:59.024+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:18:59.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:59.208+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:59.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:59.208+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:59.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:59.208+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:18:59.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:18:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:18:59.208+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:00.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:00.226+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:00.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:00.226+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:00.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:00.226+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:00.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:00.226+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:00.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:00.018+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:00.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:00.018+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:00.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:00.018+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:00.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:00.018+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:00.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:00.047+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:00.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:00.047+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:00.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:00.047+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:00.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:00.047+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:00.870 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:19:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:19:00.475+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24531 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:19:01.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:01.024+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:01.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:01.024+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:01.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:01.024+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:01.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:01.025+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:01.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:01.007+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:01.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:01.007+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:01.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:01.007+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:01.226 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:01.008+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:01.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:01.235+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:01.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:01.235+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:01.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:01.235+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:01.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:01.236+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:02.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:01.984+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:02.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:01.984+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:02.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:01.984+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:02.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:01.985+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:02.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:02.189+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:02.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:02.189+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:02.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:02.189+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:02.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:02.190+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:02.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:02.057+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:02.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:02.057+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:02.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:02.057+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:02.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:02.057+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:03.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:02.962+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:03.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:02.962+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:03.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:02.962+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:03.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:02.963+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:03.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:03.052+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:03.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:03.052+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:03.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:03.052+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:03.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:03.052+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:03.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:03.184+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:03.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:03.184+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:03.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:03.185+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:03.226 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:03.186+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:04.193 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:03.968+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:04.194 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:03.968+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:04.194 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:03.968+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:04.194 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:03.969+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:04.194 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:04.067+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:04.194 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:04.067+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:04.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:04.067+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:04.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:04.068+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:04.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:04.193+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:04.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:04.193+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:04.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:04.193+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:04.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:04.194+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:05.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:05.191+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:05.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:05.191+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:05.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:05.191+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:05.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:05.191+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:05.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:05.005+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:05.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:05.005+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:05.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:05.005+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:05.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:05.006+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:05.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:05.087+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:05.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:05.087+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:05.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:05.087+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:05.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:05.088+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:05.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:19:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:19:05.478+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24616 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:19:06.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:05.989+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:06.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:05.990+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:06.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:05.990+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:06.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:05.990+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:06.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:06.214+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:06.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:06.214+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:06.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:06.214+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:06.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:06.214+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:06.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:06.085+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:06.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:06.085+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:06.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:06.085+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:06.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:06.085+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:07.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:06.967+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:07.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:06.967+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:07.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:06.967+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:07.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:06.968+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:07.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:07.197+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:07.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:07.197+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:07.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:07.197+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:07.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:07.198+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:07.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:07.057+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:07.226 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:07.057+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:07.226 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:07.057+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:07.226 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:07.057+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:08.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:07.948+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:08.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:07.948+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:08.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:07.948+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:08.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:07.949+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:08.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:08.169+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:08.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:08.169+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:08.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:08.169+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:08.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:08.170+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:08.226 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:08.042+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:08.226 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:08.042+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:08.226 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:08.042+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:08.226 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:08.043+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:09.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:08.942+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:09.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:08.942+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:09.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:08.942+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:09.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:08.943+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:09.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:09.054+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:09.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:09.054+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:09.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:09.054+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:09.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:09.054+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:09.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:09.200+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:09.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:09.200+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:09.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:09.200+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:09.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:09.201+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:10.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:09.976+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:10.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:09.976+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:10.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:09.976+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:10.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:09.977+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:10.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:10.036+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:10.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:10.036+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:10.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:10.036+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:10.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:10.037+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:10.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:10.194+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:10.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:10.194+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:10.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:10.194+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:10.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:10.195+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:10.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:19:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:19:10.481+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24699 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:19:11.242 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:10.981+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:11.243 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:10.981+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:11.243 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:10.981+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:11.243 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:10.982+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:11.243 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:11.015+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:11.243 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:11.015+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:11.243 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:11.015+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:11.244 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:11.016+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:11.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:11.242+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:11.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:11.242+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:11.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:11.242+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:11.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:11.243+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:12.254 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:12.003+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:12.254 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:12.003+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:12.254 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:12.003+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:12.254 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:12.003+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:12.254 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:12.016+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:12.255 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:12.016+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:12.255 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:12.016+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:12.255 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:12.017+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:12.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:12.254+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:12.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:12.254+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:12.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:12.254+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:12.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:12.255+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:13.274 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:12.996+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:13.275 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:12.996+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:13.275 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:12.996+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:13.275 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:12.997+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:13.275 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:13.025+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:13.275 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:13.025+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:13.275 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:13.025+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:13.276 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:13.026+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:13.606 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:13.275+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:13.607 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:13.275+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:13.607 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:13.275+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:13.607 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:13.275+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:14.270 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:14.000+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:14.271 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:14.000+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:14.271 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:14.000+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:14.271 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:14.001+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:14.271 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:13.988+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:14.271 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:13.988+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:14.272 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:13.988+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:14.272 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:13.989+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:14.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:14.270+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:14.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:14.270+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:14.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:14.270+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:14.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:14.271+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:15.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:15.037+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:15.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:15.037+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:15.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:15.037+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:15.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:15.038+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:15.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:15.048+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:15.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:15.048+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:15.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:15.048+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:15.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:15.048+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:15.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:15.279+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:15.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:15.279+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:15.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:15.279+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:15.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:15.280+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:15.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:19:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:19:15.484+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24784 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:19:16.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:16.271+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:16.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:16.271+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:16.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:16.271+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:16.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:16.272+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:16.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:16.047+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:16.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:16.047+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:16.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:16.047+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:16.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:16.048+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:16.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:16.068+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:16.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:16.068+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:16.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:16.068+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:16.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:16.068+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:17.375 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:17.259+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:17.376 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:17.259+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:17.376 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:17.259+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:17.376 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:17.260+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:17.376 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:17.023+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:17.376 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:17.023+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:17.377 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:17.023+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:17.377 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:17.024+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:17.377 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:17.036+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:17.377 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:17.036+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:17.377 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:17.036+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:17.377 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:17.037+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:18.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:18.234+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:18.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:18.234+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:18.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:18.234+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:18.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:18.235+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:18.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:18.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:18.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:18.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:18.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:18.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:18.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:18.054+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:18.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:18.041+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:18.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:18.041+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:18.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:18.041+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:18.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:18.042+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:19.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:19.069+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:19.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:19.069+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:19.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:19.069+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:19.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:19.070+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:19.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:19.038+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:19.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:19.038+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:19.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:19.038+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:19.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:19.039+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:19.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:19.192+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:19.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:19.192+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:19.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:19.192+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:19.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:19.193+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:20.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:20.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:20.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:20.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:20.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:20.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:20.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:20.053+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:20.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:20.143+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:20.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:20.143+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:20.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:20.143+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:20.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:20.144+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:20.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:20.038+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:20.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:20.038+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:20.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:20.038+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:20.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:20.039+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:20.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:19:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:19:20.487+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24868 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:19:21.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:21.043+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:21.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:21.043+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:21.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:21.043+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:21.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:21.043+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:21.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:21.180+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:21.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:21.180+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:21.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:21.180+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:21.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:21.181+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:21.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:21.051+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:21.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:21.051+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:21.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:21.051+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:21.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:21.051+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:22.374 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:22.074+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:22.374 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:22.074+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:22.374 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:22.074+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:22.375 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:22.075+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:22.375 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:22.210+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:22.375 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:22.210+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:22.375 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:22.210+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:22.375 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:22.211+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:22.376 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:22.076+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:22.376 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:22.076+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:22.376 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:22.076+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:22.376 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:22.076+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:23.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:23.122+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:23.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:23.122+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:23.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:23.122+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:23.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:23.122+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:23.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:23.043+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:23.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:23.043+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:23.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:23.043+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:23.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:23.044+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:23.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:23.257+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:23.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:23.257+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:23.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:23.257+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:23.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:23.258+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:24.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:24.105+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:24.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:24.105+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:24.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:24.105+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:24.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:24.106+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:24.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:24.020+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:24.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:24.020+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:24.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:24.020+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:24.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:24.020+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:24.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:24.255+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:24.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:24.255+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:24.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:24.255+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:24.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:24.256+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:25.267 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:25.143+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:25.268 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:25.143+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:25.268 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:25.143+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:25.268 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:25.144+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:25.268 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:24.995+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:25.268 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:24.995+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:25.268 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:24.995+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:25.269 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:24.996+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:25.610 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:19:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:19:25.489+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24952 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:19:25.611 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:25.267+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:25.611 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:25.267+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:25.611 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:25.267+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:25.611 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:25.268+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:26.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:26.191+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:26.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:26.191+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:26.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:26.191+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:26.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:26.192+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:26.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:25.956+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:26.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:25.956+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:26.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:25.956+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:26.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:25.957+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:26.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:26.227+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:26.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:26.227+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:26.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:26.227+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:26.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:26.228+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:27.223 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:27.146+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:27.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:27.146+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:27.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:27.146+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:27.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:27.147+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:27.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:26.948+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:27.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:26.948+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:27.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:26.948+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:27.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:26.948+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:27.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:27.241+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:27.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:27.241+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:27.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:27.241+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:27.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:27.242+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:28.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:27.907+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:28.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:27.907+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:28.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:27.907+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:28.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:27.908+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:28.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:28.141+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:28.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:28.141+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:28.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:28.141+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:28.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:28.142+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:28.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:28.231+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:28.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:28.231+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:28.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:28.231+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:28.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:28.231+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:29.172 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:28.902+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:29.173 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:28.902+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:29.173 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:28.902+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:29.173 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:28.902+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:29.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:29.173+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:29.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:29.173+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:29.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:29.173+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:29.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:29.173+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:29.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:29.236+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:29.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:29.236+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:29.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:29.236+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:29.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:29.238+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:30.181 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:29.880+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:30.182 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:29.880+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:30.182 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:29.880+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:30.182 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:29.881+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:30.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:30.181+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:30.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:30.181+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:30.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:30.181+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:30.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:30.182+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:30.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:30.273+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:30.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:30.273+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:30.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:30.273+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:30.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:30.274+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:30.859 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:19:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:19:30.492+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25036 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:19:31.176 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:30.859+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:31.177 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:30.859+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:31.177 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:30.859+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:31.177 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:30.860+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:31.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:31.177+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:31.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:31.177+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:31.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:31.177+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:31.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:31.177+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:31.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:31.293+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:31.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:31.293+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:31.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:31.293+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:31.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:31.294+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:32.215 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:31.853+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:32.215 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:31.853+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:32.216 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:31.853+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:32.216 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:31.854+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:32.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:32.215+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:32.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:32.215+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:32.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:32.215+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:32.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:32.216+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:32.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:32.343+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:32.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:32.344+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:32.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:32.344+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:32.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:32.344+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:33.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:32.899+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:33.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:32.899+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:33.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:32.899+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:33.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:32.900+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:33.607 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:33.379+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:33.607 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:33.379+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:33.607 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:33.379+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:33.607 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:33.380+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:33.607 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:33.237+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:33.608 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:33.237+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:33.608 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:33.237+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:33.608 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:33.237+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:34.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:33.882+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:34.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:33.882+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:34.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:33.883+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:34.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:33.883+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:34.662 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:34.271+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:34.662 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:34.271+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:34.662 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:34.271+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:34.663 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:34.272+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:34.663 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:34.368+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:34.663 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:34.368+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:34.663 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:34.368+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:34.663 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:34.369+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:34.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:34.871+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:34.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:34.871+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:34.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:34.871+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:34.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:34.872+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:35.619 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:19:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:19:35.495+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25121 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:19:35.619 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:35.273+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:35.620 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:35.273+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:35.620 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:35.273+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:35.620 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:35.274+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:35.620 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:35.333+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:35.620 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:35.333+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:35.621 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:35.333+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:35.621 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:35.334+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:36.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:35.881+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:36.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:35.881+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:36.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:35.881+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:36.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:35.882+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:36.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:36.366+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:36.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:36.366+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:36.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:36.366+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:36.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:36.367+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:36.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:36.256+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:36.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:36.256+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:36.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:36.256+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:36.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:36.257+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:37.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:36.891+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:37.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:36.891+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:37.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:36.891+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:37.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:36.892+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:37.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:37.392+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:37.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:37.392+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:37.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:37.392+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:37.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:37.392+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:37.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:37.274+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:37.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:37.274+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:37.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:37.274+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:37.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:37.274+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:38.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:37.846+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:38.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:37.846+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:38.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:37.846+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:38.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:37.847+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:38.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:38.286+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:38.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:38.286+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:38.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:38.286+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:38.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:38.287+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:38.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:38.363+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:38.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:38.363+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:38.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:38.363+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:38.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:38.363+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:39.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:38.817+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:39.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:38.817+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:39.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:38.817+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:39.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:38.818+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:39.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:39.335+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:39.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:39.335+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:39.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:39.335+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:39.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:39.336+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:39.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:39.245+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:39.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:39.245+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:39.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:39.245+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:39.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:39.246+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:40.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:39.777+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:40.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:39.777+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:40.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:39.777+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:40.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:39.777+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:40.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:40.229+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:40.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:40.229+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:40.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:40.229+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:40.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:40.230+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:40.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:40.290+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:40.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:40.290+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:40.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:40.290+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:40.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:40.291+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:40.806 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:19:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:19:40.497+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25205 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:19:41.205 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:40.809+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:41.205 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:40.809+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:41.206 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:40.809+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:41.206 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:40.809+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:41.473 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:41.310+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:41.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:41.310+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:41.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:41.310+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:41.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:41.310+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:41.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:41.205+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:41.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:41.205+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:41.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:41.205+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:41.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:41.206+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:42.160 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:41.849+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:42.160 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:41.849+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:42.160 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:41.849+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:42.160 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:41.850+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:42.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:42.160+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:42.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:42.160+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:42.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:42.160+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:42.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:42.161+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:42.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:42.262+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:42.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:42.262+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:42.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:42.262+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:42.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:42.263+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:43.159 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:42.840+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:43.159 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:42.840+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:43.159 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:42.840+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:43.160 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:42.841+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:43.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:43.159+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:43.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:43.159+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:43.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:43.159+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:43.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:43.160+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:43.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:43.232+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:43.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:43.232+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:43.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:43.232+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:43.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:43.233+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:43.878 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:43.837+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:43.878 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:43.837+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:43.878 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:43.837+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:43.878 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:43.838+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:44.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:44.181+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:44.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:44.181+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:44.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:44.181+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:44.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:44.181+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:44.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:44.215+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:44.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:44.215+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:44.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:44.215+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:44.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:44.216+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:45.189 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:44.832+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:45.190 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:44.832+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:45.190 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:44.832+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:45.190 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:44.833+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:45.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:45.189+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:45.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:45.189+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:45.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:45.189+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:45.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:45.191+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:45.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:45.263+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:45.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:45.263+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:45.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:45.263+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:45.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:45.264+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:45.808 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:19:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:19:45.500+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25290 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:19:46.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:45.809+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:46.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:45.809+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:46.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:45.809+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:46.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:45.810+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:46.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:46.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:46.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:46.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:46.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:46.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:46.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:46.241+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:46.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:46.243+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:46.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:46.243+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:46.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:46.243+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:46.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:46.243+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:47.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:46.858+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:47.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:46.858+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:47.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:46.858+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:47.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:46.859+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:47.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:47.290+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:47.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:47.290+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:47.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:47.290+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:47.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:47.291+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:47.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:47.250+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:47.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:47.250+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:47.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:47.250+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:47.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:47.251+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:48.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:47.860+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:48.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:47.860+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:48.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:47.860+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:48.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:47.860+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:48.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:48.278+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:48.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:48.278+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:48.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:48.278+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:48.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:48.280+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:48.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:48.282+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:48.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:48.282+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:48.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:48.282+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:48.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:48.282+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:49.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:48.859+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:49.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:48.859+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:49.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:48.859+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:49.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:48.860+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:49.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:49.255+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:49.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:49.255+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:49.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:49.255+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:49.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:49.256+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:49.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:49.260+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:49.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:49.260+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:49.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:49.260+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:49.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:49.261+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:50.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:49.854+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:50.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:49.854+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:50.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:49.854+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:50.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:49.855+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:50.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:50.295+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:50.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:50.295+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:50.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:50.295+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:50.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:50.296+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:50.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:50.276+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:50.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:50.276+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:50.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:50.276+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:50.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:50.277+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:50.861 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:19:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:19:50.503+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25372 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:19:51.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:50.862+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:51.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:50.862+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:51.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:50.862+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:51.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:50.863+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:51.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:51.343+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:51.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:51.343+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:51.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:51.343+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:51.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:51.344+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:51.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:51.242+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:51.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:51.242+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:51.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:51.242+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:51.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:51.242+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:52.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:51.869+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:52.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:51.869+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:52.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:51.869+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:52.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:51.869+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:52.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:52.389+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:52.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:52.389+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:52.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:52.389+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:52.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:52.389+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:52.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:52.257+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:52.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:52.257+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:52.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:52.257+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:52.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:52.257+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:53.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:52.882+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:53.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:52.882+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:53.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:52.882+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:53.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:52.883+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:53.608 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:53.385+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:53.608 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:53.385+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:53.609 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:53.385+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:53.609 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:53.386+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:53.609 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:53.259+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:53.609 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:53.259+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:53.609 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:53.259+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:53.610 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:53.259+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:53.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:53.836+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:53.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:53.836+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:53.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:53.836+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:53.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:53.836+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:54.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:54.357+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:54.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:54.357+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:54.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:54.357+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:54.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:54.358+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:54.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:54.285+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:54.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:54.285+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:54.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:54.285+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:54.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:54.286+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:55.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:54.856+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:55.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:54.856+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:55.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:54.856+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:55.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:54.856+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:55.506 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:55.392+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:55.506 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:55.392+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:55.506 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:55.392+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:55.506 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:55.393+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:55.506 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:55.253+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:55.507 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:55.253+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:55.507 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:55.253+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:55.507 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:55.253+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:55.899 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:19:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:19:55.506+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25457 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:19:56.167 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:55.899+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:56.167 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:55.899+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:56.167 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:55.899+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:56.167 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:55.900+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:56.440 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:56.243+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:56.440 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:56.243+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:56.440 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:56.243+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:56.441 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:56.244+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:56.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:56.440+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:56.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:56.440+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:56.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:56.440+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:56.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:56.441+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:57.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:56.948+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:57.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:56.948+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:57.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:56.948+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:57.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:56.949+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:57.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:57.472+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:57.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:57.472+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:57.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:57.472+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:57.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:57.473+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:57.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:57.292+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:57.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:57.292+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:57.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:57.292+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:57.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:57.293+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:58.323 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:57.988+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:58.324 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:57.988+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:58.324 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:57.988+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:58.324 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:57.988+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:58.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:58.323+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:58.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:58.323+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:58.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:58.323+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:58.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:58.324+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:19:58.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:58.459+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:58.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:58.459+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:58.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:58.459+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:58.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:58.459+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:59.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:58.953+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:59.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:58.953+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:59.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:58.953+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:19:59.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:58.954+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:59.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:59.474+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:59.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:59.474+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:59.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:59.474+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:19:59.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:19:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:19:59.475+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:19:59.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:59.334+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:59.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:59.334+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:59.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:59.334+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:19:59.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:19:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:19:59.335+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:00.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:59.915+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:00.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:59.915+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:00.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:59.915+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:00.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:19:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:19:59.916+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:00.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:00.441+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:00.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:00.441+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:00.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:00.441+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:00.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:00.442+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:00.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:00.323+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:00.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:00.323+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:00.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:00.323+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:00.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:00.324+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:00.931 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:20:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:20:00.508+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25539 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:20:01.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:00.932+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:01.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:00.932+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:01.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:00.932+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:01.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:00.932+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:01.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:01.396+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:01.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:01.396+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:01.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:01.396+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:01.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:01.396+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:01.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:01.341+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:01.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:01.341+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:01.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:01.341+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:01.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:01.342+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:02.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:01.947+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:02.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:01.947+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:02.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:01.947+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:02.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:01.948+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:02.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:02.398+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:02.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:02.398+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:02.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:02.398+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:02.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:02.400+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:02.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:02.330+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:02.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:02.330+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:02.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:02.330+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:02.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:02.331+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:03.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:02.917+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:03.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:02.917+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:03.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:02.917+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:03.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:02.918+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:03.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:03.417+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:03.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:03.417+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:03.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:03.417+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:03.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:03.418+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:03.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:03.361+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:03.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:03.361+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:03.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:03.361+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:03.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:03.361+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:04.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:03.877+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:04.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:03.877+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:04.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:03.877+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:04.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:03.877+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:04.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:04.411+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:04.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:04.411+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:04.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:04.411+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:04.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:04.412+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:04.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:04.431+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:04.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:04.431+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:04.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:04.431+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:04.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:04.432+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:05.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:04.905+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:05.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:04.905+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:05.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:04.905+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:05.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:04.906+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:05.634 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:20:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:20:05.510+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25623 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:20:05.635 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:05.460+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:05.635 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:05.460+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:05.635 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:05.460+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:05.635 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:05.461+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:05.635 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:05.400+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:05.636 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:05.400+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:05.636 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:05.400+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:05.636 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:05.401+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:05.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:05.866+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:05.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:05.866+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:05.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:05.866+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:05.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:05.866+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:06.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:06.372+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:06.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:06.372+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:06.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:06.372+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:06.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:06.374+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:06.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:06.500+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:06.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:06.500+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:06.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:06.500+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:06.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:06.501+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:07.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:06.879+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:07.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:06.879+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:07.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:06.879+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:07.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:06.880+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:07.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:07.410+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:07.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:07.410+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:07.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:07.410+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:07.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:07.411+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:07.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:07.550+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:07.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:07.550+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:07.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:07.550+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:07.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:07.551+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:08.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:07.834+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:08.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:07.834+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:08.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:07.834+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:08.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:07.835+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:08.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:08.416+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:08.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:08.416+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:08.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:08.416+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:08.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:08.417+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:08.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:08.507+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:08.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:08.507+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:08.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:08.507+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:08.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:08.508+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:09.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:08.833+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:09.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:08.833+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:09.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:08.833+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:09.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:08.834+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:09.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:09.488+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:09.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:09.488+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:09.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:09.488+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:09.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:09.488+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:09.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:09.421+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:09.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:09.421+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:09.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:09.421+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:09.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:09.422+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:10.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:09.790+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:10.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:09.790+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:10.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:09.790+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:10.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:09.791+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:10.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:10.373+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:10.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:10.373+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:10.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:10.373+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:10.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:10.374+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:10.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:20:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:20:10.513+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25707 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:20:10.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:10.524+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:10.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:10.524+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:10.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:10.524+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:10.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:10.525+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:10.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:10.762+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:10.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:10.762+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:10.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:10.762+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:10.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:10.763+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:11.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:11.479+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:11.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:11.479+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:11.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:11.479+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:11.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:11.479+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:11.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:11.380+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:11.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:11.380+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:11.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:11.380+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:11.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:11.381+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:12.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:11.763+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:12.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:11.763+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:12.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:11.763+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:12.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:11.763+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:12.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:12.408+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:12.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:12.408+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:12.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:12.408+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:12.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:12.409+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:12.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:12.463+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:12.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:12.463+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:12.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:12.463+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:12.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:12.464+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:13.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:12.776+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:13.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:12.776+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:13.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:12.776+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:13.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:12.777+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:13.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:13.401+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:13.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:13.401+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:13.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:13.401+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:13.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:13.401+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:13.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:13.498+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:13.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:13.498+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:13.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:13.498+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:13.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:13.499+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:14.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:13.826+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:14.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:13.826+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:14.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:13.826+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:14.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:13.826+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:14.530 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:14.433+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:14.531 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:14.433+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:14.531 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:14.433+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:14.531 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:14.434+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:14.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:14.528+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:14.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:14.528+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:14.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:14.528+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:14.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:14.529+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:15.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:14.857+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:15.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:14.857+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:15.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:14.857+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:15.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:14.858+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:15.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:20:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:20:15.516+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25792 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:20:15.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:15.508+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:15.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:15.508+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:15.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:15.508+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:15.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:15.510+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:15.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:15.434+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:15.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:15.434+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:15.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:15.434+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:15.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:15.435+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:16.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:15.848+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:16.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:15.848+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:16.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:15.848+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:16.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:15.849+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:16.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:16.436+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:16.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:16.436+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:16.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:16.436+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:16.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:16.436+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:16.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:16.550+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:16.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:16.550+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:16.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:16.550+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:16.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:16.551+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:17.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:16.878+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:17.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:16.878+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:17.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:16.878+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:17.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:16.878+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:17.515 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:17.395+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:17.516 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:17.395+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:17.516 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:17.395+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:17.516 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:17.396+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:17.926 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:17.514+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:17.927 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:17.514+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:17.927 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:17.515+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:17.927 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:17.516+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:18.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:17.925+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:18.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:17.925+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:18.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:17.925+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:18.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:17.926+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:18.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:18.504+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:18.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:18.504+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:18.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:18.504+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:18.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:18.505+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:18.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:18.361+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:18.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:18.361+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:18.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:18.361+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:18.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:18.362+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:19.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:18.913+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:19.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:18.913+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:19.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:18.913+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:19.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:18.914+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:19.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:19.473+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:19.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:19.473+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:19.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:19.473+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:19.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:19.473+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:19.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:19.349+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:19.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:19.349+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:19.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:19.349+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:19.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:19.349+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:20.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:19.887+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:20.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:19.887+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:20.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:19.887+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:20.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:19.887+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:20.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:20.387+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:20.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:20.387+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:20.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:20.387+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:20.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:20.388+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:20.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:20.426+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:20.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:20.426+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:20.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:20.426+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:20.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:20.426+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:20.892 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:20:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:20:20.519+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25876 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:20:21.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:20.892+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:21.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:20.892+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:21.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:20.892+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:21.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:20.892+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:21.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:21.378+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:21.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:21.378+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:21.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:21.378+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:21.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:21.379+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:21.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:21.390+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:21.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:21.390+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:21.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:21.390+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:21.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:21.391+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:22.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:21.875+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:22.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:21.875+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:22.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:21.875+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:22.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:21.876+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:22.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:22.387+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:22.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:22.387+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:22.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:22.387+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:22.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:22.388+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:22.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:22.411+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:22.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:22.411+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:22.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:22.411+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:22.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:22.412+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:23.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:22.832+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:23.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:22.832+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:23.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:22.832+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:23.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:22.833+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:23.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:23.416+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:23.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:23.416+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:23.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:23.416+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:23.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:23.417+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:23.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:23.383+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:23.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:23.383+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:23.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:23.383+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:23.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:23.383+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:24.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:23.796+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:24.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:23.796+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:24.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:23.796+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:24.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:23.797+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:24.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:24.362+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:24.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:24.362+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:24.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:24.362+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:24.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:24.364+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:24.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:24.373+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:24.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:24.373+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:24.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:24.373+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:24.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:24.373+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:25.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:24.838+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:25.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:24.838+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:25.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:24.838+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:25.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:24.838+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:25.647 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:20:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:20:25.522+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25961 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:20:25.647 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:25.390+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:25.647 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:25.390+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:25.648 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:25.390+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:25.648 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:25.391+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:25.648 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:25.356+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:25.648 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:25.356+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:25.648 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:25.356+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:25.648 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:25.356+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:25.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:25.843+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:25.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:25.843+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:25.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:25.843+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:25.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:25.844+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:26.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:26.373+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:26.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:26.373+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:26.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:26.373+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:26.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:26.373+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:26.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:26.428+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:26.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:26.428+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:26.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:26.428+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:26.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:26.429+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:27.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:26.803+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:27.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:26.803+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:27.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:26.803+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:27.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:26.804+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:27.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:27.411+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:27.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:27.411+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:27.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:27.411+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:27.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:27.412+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:27.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:27.411+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:27.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:27.411+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:27.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:27.411+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:27.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:27.412+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:28.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:27.823+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:28.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:27.823+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:28.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:27.823+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:28.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:27.824+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:28.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:28.439+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:28.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:28.439+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:28.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:28.439+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:28.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:28.439+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:28.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:28.435+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:28.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:28.435+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:28.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:28.435+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:28.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:28.435+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:29.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:28.776+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:29.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:28.776+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:29.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:28.776+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:29.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:28.777+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:29.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:29.406+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:29.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:29.407+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:29.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:29.407+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:29.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:29.407+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:29.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:29.422+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:29.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:29.422+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:29.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:29.422+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:29.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:29.422+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:30.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:29.761+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:30.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:29.761+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:30.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:29.761+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:30.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:29.762+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:30.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:30.432+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:30.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:30.432+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:30.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:30.432+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:30.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:30.433+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:30.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:30.420+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:30.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:30.420+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:30.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:30.420+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:30.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:30.421+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:30.788 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:20:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:20:30.525+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26045 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:20:31.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:30.788+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:31.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:30.788+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:31.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:30.788+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:31.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:30.789+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:31.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:31.395+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:31.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:31.395+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:31.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:31.396+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:31.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:31.396+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:31.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:31.455+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:31.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:31.455+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:31.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:31.455+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:31.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:31.456+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:32.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:31.750+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:32.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:31.750+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:32.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:31.750+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:32.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:31.751+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:32.715 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:32.409+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:32.716 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:32.409+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:32.716 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:32.409+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:32.716 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:32.410+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:32.716 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:32.468+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:32.717 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:32.468+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:32.717 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:32.468+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:32.717 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:32.469+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:32.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:32.713+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:32.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:32.713+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:32.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:32.713+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:32.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:32.713+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:33.691 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:33.393+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:33.692 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:33.393+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:33.692 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:33.393+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:33.692 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:33.393+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:33.692 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:33.427+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:33.692 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:33.427+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:33.692 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:33.427+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:33.693 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:33.427+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:33.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:33.691+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:33.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:33.691+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:33.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:33.691+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:33.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:33.692+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:34.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:34.404+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:34.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:34.404+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:34.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:34.404+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:34.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:34.405+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:34.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:34.430+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:34.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:34.430+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:34.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:34.430+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:34.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:34.431+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:35.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:34.742+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:35.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:34.742+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:35.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:34.742+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:35.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:34.742+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:35.652 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:20:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:20:35.528+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26129 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:20:35.653 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:35.381+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:35.653 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:35.381+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:35.653 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:35.381+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:35.654 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:35.382+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:35.654 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:35.447+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:35.654 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:35.447+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:35.655 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:35.447+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:35.655 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:35.448+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:35.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:35.749+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:35.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:35.749+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:35.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:35.749+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:35.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:35.750+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:36.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:36.417+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:36.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:36.417+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:36.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:36.417+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:36.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:36.418+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:36.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:36.476+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:36.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:36.476+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:36.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:36.476+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:36.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:36.477+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:37.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:36.743+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:37.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:36.743+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:37.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:36.743+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:37.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:36.743+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:37.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:37.382+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:37.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:37.382+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:37.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:37.382+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:37.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:37.384+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:37.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:37.444+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:37.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:37.444+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:37.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:37.444+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:37.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:37.445+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:38.034 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:37.726+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:38.034 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:37.726+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:38.034 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:37.726+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:38.034 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:37.726+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:38.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:38.409+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:38.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:38.409+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:38.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:38.409+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:38.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:38.410+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:38.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:38.491+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:38.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:38.491+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:38.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:38.491+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:38.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:38.492+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:39.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:38.749+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:39.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:38.749+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:39.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:38.749+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:39.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:38.750+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:39.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:39.434+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:39.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:39.434+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:39.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:39.434+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:39.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:39.435+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:39.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:39.476+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:39.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:39.476+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:39.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:39.476+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:39.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:39.477+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:40.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:39.728+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:40.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:39.728+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:40.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:39.728+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:40.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:39.729+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:40.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:40.391+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:40.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:40.391+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:40.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:40.391+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:40.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:40.392+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:40.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:20:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:20:40.531+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26212 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:20:40.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:40.515+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:40.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:40.515+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:40.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:40.515+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:40.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:40.516+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:40.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:40.698+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:40.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:40.698+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:40.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:40.698+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:40.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:40.698+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:41.697 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:41.529+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:41.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:41.529+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:41.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:41.529+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:41.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:41.530+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:41.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:41.425+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:41.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:41.425+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:41.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:41.425+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:41.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:41.426+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:41.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:41.697+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:41.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:41.697+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:41.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:41.697+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:41.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:41.698+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:42.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:42.390+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:42.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:42.390+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:42.574 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:42.390+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:42.574 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:42.391+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:42.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:42.573+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:42.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:42.573+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:42.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:42.573+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:42.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:42.574+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:42.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:42.697+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:42.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:42.697+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:42.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:42.697+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:42.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:42.697+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:43.688 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:43.533+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:43.688 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:43.533+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:43.689 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:43.533+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:43.689 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:43.534+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:43.689 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:43.380+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:43.689 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:43.380+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:43.689 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:43.380+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:43.689 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:43.381+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:43.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:43.688+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:43.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:43.688+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:43.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:43.688+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:43.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:43.689+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:44.723 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:44.551+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:44.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:44.551+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:44.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:44.551+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:44.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:44.552+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:44.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:44.413+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:44.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:44.413+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:44.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:44.413+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:44.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:44.413+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:44.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:44.641+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:44.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:44.641+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:44.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:44.641+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:44.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:44.642+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:45.598 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:20:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:20:45.534+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26297 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:20:45.598 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:45.405+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:45.598 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:45.405+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:45.598 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:45.405+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:45.599 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:45.406+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:45.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:45.598+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:45.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:45.598+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:45.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:45.598+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:45.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:45.599+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:45.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:45.644+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:45.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:45.644+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:45.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:45.644+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:45.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:45.644+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:46.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:46.566+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:46.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:46.566+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:46.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:46.566+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:46.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:46.566+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:46.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:46.418+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:46.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:46.418+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:46.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:46.418+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:46.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:46.419+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:46.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:46.630+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:46.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:46.630+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:46.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:46.630+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:46.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:46.630+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:47.625 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:47.580+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:47.625 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:47.580+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:47.625 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:47.580+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:47.625 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:47.581+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:47.625 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:47.373+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:47.626 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:47.373+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:47.626 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:47.373+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:47.626 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:47.374+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:47.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:47.678+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:47.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:47.678+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:47.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:47.678+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:47.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:47.679+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:48.625 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:48.374+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:48.625 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:48.374+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:48.625 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:48.375+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:48.625 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:48.375+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:48.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:48.625+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:48.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:48.625+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:48.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:48.625+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:48.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:48.626+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:48.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:48.668+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:48.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:48.668+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:48.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:48.668+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:48.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:48.669+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:49.606 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:49.411+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:49.607 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:49.411+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:49.607 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:49.411+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:49.607 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:49.411+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:49.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:49.633+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:49.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:49.633+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:49.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:49.633+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:49.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:49.634+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:49.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:49.606+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:49.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:49.606+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:49.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:49.606+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:49.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:49.607+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:50.723 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:20:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:20:50.537+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26379 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:20:50.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:50.560+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:50.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:50.560+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:50.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:50.560+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:50.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:50.561+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:50.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:50.461+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:50.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:50.461+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:50.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:50.461+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:50.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:50.462+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:50.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:50.588+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:50.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:50.588+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:50.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:50.588+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:50.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:50.589+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:51.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:51.450+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:51.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:51.450+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:51.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:51.450+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:51.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:51.451+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:51.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:51.557+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:51.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:51.557+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:51.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:51.557+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:51.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:51.558+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:51.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:51.567+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:51.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:51.567+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:51.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:51.567+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:51.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:51.568+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:52.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:52.426+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:52.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:52.426+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:52.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:52.426+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:52.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:52.427+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:52.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:52.536+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:52.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:52.536+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:52.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:52.536+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:52.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:52.536+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:52.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:52.524+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:52.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:52.524+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:52.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:52.524+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:52.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:52.525+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:53.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:53.385+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:53.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:53.385+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:53.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:53.385+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:53.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:53.386+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:53.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:53.520+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:53.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:53.520+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:53.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:53.520+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:53.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:53.521+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:53.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:53.584+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:53.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:53.584+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:53.726 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:53.584+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:53.726 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:53.585+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:54.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:54.414+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:54.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:54.414+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:54.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:54.414+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:54.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:54.415+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:54.549+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:54.549+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:54.549+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:54.550+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:54.726 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:54.555+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:54.726 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:54.555+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:54.726 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:54.555+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:54.726 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:54.555+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:55.662 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:20:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:20:55.540+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26464 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:20:55.663 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:55.600+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:55.663 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:55.600+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:55.663 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:55.600+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:55.663 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:55.601+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:55.663 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:55.387+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:55.664 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:55.387+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:55.664 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:55.387+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:55.664 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:55.388+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:55.664 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:55.534+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:55.664 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:55.534+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:55.664 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:55.534+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:55.665 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:55.535+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:56.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:56.401+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:56.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:56.401+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:56.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:56.401+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:56.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:56.402+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:56.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:56.594+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:56.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:56.594+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:56.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:56.594+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:56.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:56.595+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:56.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:56.578+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:56.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:56.578+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:56.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:56.578+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:56.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:56.579+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:57.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:57.381+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:57.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:57.381+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:57.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:57.381+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:57.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:57.382+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:57.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:57.616+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:57.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:57.616+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:57.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:57.616+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:57.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:57.617+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:57.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:57.564+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:57.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:57.564+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:57.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:57.564+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:57.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:57.565+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:58.636 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:58.550+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:58.636 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:58.550+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:58.636 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:58.550+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:58.637 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:58.550+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:58.637 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:58.375+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:58.637 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:58.375+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:58.637 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:58.375+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:58.637 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:58.376+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:58.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:58.636+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:58.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:58.636+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:58.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:58.636+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:58.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:58.637+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:59.546 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:59.393+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:59.546 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:59.393+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:59.546 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:59.393+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:20:59.546 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:20:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:20:59.394+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:20:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:59.668+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:59.668+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:59.668+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:20:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:20:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:20:59.669+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:20:59.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:59.546+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:59.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:59.546+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:59.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:59.546+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:20:59.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:20:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:20:59.546+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:00.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:00.391+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:00.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:00.391+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:00.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:00.391+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:00.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:00.392+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:00.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:21:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:21:00.543+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26547 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:21:00.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:00.646+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:00.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:00.646+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:00.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:00.646+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:00.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:00.647+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:00.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:00.556+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:00.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:00.556+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:00.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:00.556+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:00.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:00.556+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:01.691 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:01.401+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:01.691 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:01.401+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:01.691 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:01.401+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:01.691 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:01.402+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:01.691 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:01.535+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:01.692 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:01.535+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:01.692 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:01.535+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:01.692 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:01.536+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:01.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:01.691+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:01.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:01.691+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:01.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:01.691+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:01.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:01.692+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:02.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:02.376+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:02.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:02.376+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:02.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:02.376+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:02.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:02.377+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:02.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:02.583+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:02.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:02.583+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:02.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:02.583+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:02.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:02.583+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:03.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:02.737+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:03.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:02.737+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:03.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:02.737+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:03.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:02.737+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:03.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:03.405+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:03.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:03.405+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:03.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:03.405+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:03.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:03.406+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:03.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:03.550+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:03.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:03.550+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:03.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:03.550+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:03.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:03.550+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:04.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:03.768+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:04.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:03.768+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:04.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:03.768+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:04.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:03.769+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:04.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:04.357+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:04.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:04.357+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:04.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:04.357+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:04.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:04.358+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:04.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:04.562+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:04.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:04.562+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:04.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:04.562+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:04.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:04.563+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:05.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:04.736+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:05.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:04.736+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:05.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:04.736+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:05.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:04.737+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:05.668 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:21:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:21:05.546+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26632 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:21:05.669 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:05.550+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:05.669 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:05.550+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:05.669 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:05.550+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:05.669 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:05.551+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:05.670 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:05.386+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:05.670 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:05.386+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:05.670 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:05.386+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:05.670 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:05.387+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:05.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:05.739+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:05.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:05.739+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:05.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:05.739+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:05.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:05.740+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:06.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:06.576+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:06.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:06.576+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:06.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:06.576+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:06.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:06.577+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:06.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:06.346+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:06.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:06.346+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:06.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:06.346+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:06.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:06.346+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:07.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:06.724+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:07.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:06.724+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:07.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:06.724+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:07.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:06.725+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:07.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:07.395+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:07.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:07.395+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:07.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:07.395+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:07.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:07.395+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:07.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:07.573+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:07.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:07.573+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:07.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:07.573+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:07.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:07.574+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:08.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:07.750+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:08.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:07.750+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:08.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:07.750+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:08.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:07.751+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:08.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:08.360+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:08.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:08.360+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:08.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:08.360+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:08.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:08.361+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:08.703 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:08.539+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:08.703 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:08.539+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:08.703 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:08.539+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:08.703 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:08.540+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:08.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:08.702+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:08.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:08.702+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:08.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:08.702+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:08.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:08.703+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:09.580 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:09.392+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:09.580 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:09.392+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:09.580 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:09.392+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:09.580 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:09.393+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:09.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:09.737+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:09.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:09.737+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:09.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:09.737+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:09.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:09.737+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:09.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:09.580+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:09.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:09.580+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:09.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:09.580+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:09.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:09.580+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:10.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:10.354+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:10.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:10.354+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:10.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:10.354+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:10.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:10.355+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:10.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:21:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:21:10.549+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26716 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:21:10.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:10.779+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:10.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:10.779+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:10.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:10.779+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:10.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:10.780+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:10.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:10.577+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:10.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:10.577+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:10.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:10.577+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:10.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:10.577+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:11.622 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:11.332+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:11.622 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:11.332+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:11.622 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:11.332+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:11.622 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:11.333+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:11.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:11.747+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:11.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:11.747+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:11.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:11.747+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:11.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:11.748+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:11.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:11.622+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:11.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:11.622+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:11.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:11.622+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:11.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:11.623+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:12.643 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:12.361+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:12.644 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:12.361+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:12.644 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:12.361+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:12.644 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:12.362+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:12.967 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:12.756+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:12.967 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:12.756+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:12.967 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:12.756+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:12.967 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:12.757+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:12.967 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:12.643+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:12.968 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:12.643+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:12.968 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:12.643+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:12.968 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:12.644+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:13.611 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:13.350+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:13.612 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:13.350+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:13.612 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:13.350+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:13.612 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:13.351+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:13.973 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:13.741+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:13.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:13.741+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:13.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:13.741+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:13.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:13.741+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:13.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:13.612+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:13.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:13.612+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:13.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:13.612+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:13.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:13.612+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:14.637 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:14.370+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:14.637 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:14.370+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:14.637 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:14.370+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:14.637 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:14.370+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:14.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:14.750+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:14.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:14.750+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:14.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:14.750+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:14.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:14.751+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:14.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:14.637+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:14.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:14.637+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:14.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:14.637+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:14.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:14.637+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:15.654 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:21:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:21:15.552+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26801 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:21:15.655 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:15.380+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:15.655 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:15.380+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:15.655 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:15.380+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:15.655 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:15.381+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:15.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:15.797+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:15.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:15.797+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:15.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:15.797+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:15.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:15.798+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:15.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:15.654+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:15.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:15.654+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:15.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:15.654+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:15.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:15.655+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:16.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:16.364+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:16.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:16.364+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:16.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:16.364+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:16.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:16.365+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:16.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:16.833+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:16.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:16.833+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:16.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:16.833+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:16.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:16.834+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:16.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:16.631+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:16.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:16.631+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:16.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:16.631+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:16.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:16.631+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:17.608 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:17.354+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:17.608 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:17.354+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:17.609 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:17.354+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:17.609 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:17.355+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:17.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:17.608+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:17.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:17.608+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:17.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:17.608+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:17.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:17.609+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:17.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:17.842+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:17.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:17.842+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:17.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:17.842+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:17.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:17.843+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:18.641 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:18.367+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:18.641 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:18.367+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:18.641 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:18.367+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:18.641 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:18.367+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:18.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:18.881+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:18.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:18.881+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:18.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:18.881+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:18.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:18.882+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:18.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:18.641+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:18.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:18.641+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:18.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:18.641+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:18.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:18.642+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:19.654 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:19.342+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:19.655 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:19.342+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:19.655 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:19.342+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:19.655 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:19.343+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:19.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:19.861+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:19.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:19.861+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:19.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:19.861+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:19.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:19.862+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:19.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:19.654+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:19.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:19.654+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:19.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:19.654+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:19.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:19.655+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:20.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:20.335+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:20.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:20.335+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:20.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:20.335+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:20.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:20.335+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:20.902 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:21:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:21:20.555+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26885 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:21:20.903 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:20.640+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:20.903 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:20.641+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:20.903 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:20.641+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:20.903 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:20.641+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:21.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:20.902+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:21.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:20.902+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:21.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:20.902+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:21.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:20.903+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:21.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:21.620+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:21.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:21.620+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:21.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:21.620+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:21.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:21.621+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:21.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:21.374+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:21.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:21.374+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:21.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:21.374+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:21.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:21.375+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:22.183 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:21.857+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:22.183 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:21.857+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:22.183 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:21.857+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:22.183 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:21.858+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:22.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:22.420+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:22.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:22.420+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:22.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:22.420+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:22.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:22.421+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:22.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:22.850+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:22.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:22.850+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:22.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:22.850+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:22.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:22.851+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:22.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:22.657+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:22.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:22.657+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:22.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:22.657+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:22.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:22.657+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:23.669 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:23.409+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:23.669 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:23.409+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:23.669 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:23.409+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:23.669 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:23.410+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:23.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:23.668+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:23.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:23.668+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:23.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:23.668+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:23.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:23.668+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:23.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:23.809+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:23.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:23.809+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:23.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:23.809+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:23.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:23.810+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:24.717 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:24.449+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:24.717 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:24.449+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:24.717 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:24.449+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:24.718 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:24.450+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:24.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:24.716+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:24.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:24.716+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:24.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:24.716+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:24.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:24.717+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:24.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:24.763+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:24.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:24.763+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:24.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:24.763+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:24.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:24.764+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:25.557 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:25.478+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:25.558 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:25.478+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:25.558 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:25.478+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:25.558 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:25.479+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:25.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:21:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:21:25.556+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26968 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:21:25.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:25.762+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:25.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:25.762+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:25.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:25.762+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:25.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:25.763+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:25.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:25.781+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:25.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:25.781+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:25.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:25.781+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:25.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:25.782+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:26.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:26.527+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:26.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:26.527+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:26.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:26.527+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:26.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:26.528+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:27.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:26.825+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:27.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:26.825+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:27.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:26.825+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:27.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:26.825+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:27.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:26.799+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:27.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:26.799+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:27.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:26.799+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:27.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:26.799+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:27.846 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:27.551+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:27.847 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:27.551+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:27.847 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:27.551+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:27.847 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:27.552+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:27.847 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:27.786+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:27.847 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:27.786+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:27.847 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:27.786+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:27.847 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:27.787+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:28.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:27.845+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:28.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:27.845+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:28.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:27.845+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:28.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:27.846+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:28.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:28.535+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:28.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:28.535+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:28.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:28.535+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:28.823 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:28.537+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:28.823 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:28.777+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:28.823 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:28.777+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:28.823 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:28.777+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:28.823 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:28.778+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:29.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:28.821+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:29.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:28.821+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:29.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:28.821+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:29.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:28.822+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:29.495 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:29.493+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:29.495 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:29.493+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:29.495 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:29.493+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:29.495 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:29.494+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:29.783 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:29.734+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:29.784 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:29.734+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:29.784 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:29.734+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:29.784 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:29.735+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:30.203 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:29.782+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:30.203 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:29.782+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:30.203 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:29.782+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:30.203 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:29.783+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:30.807 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:21:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:21:30.559+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27052 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:21:30.807 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:30.761+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:30.807 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:30.761+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:30.807 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:30.761+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:30.808 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:30.761+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:30.808 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:30.520+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:30.808 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:30.520+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:30.808 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:30.520+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:30.808 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:30.521+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:31.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:30.806+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:31.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:30.806+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:31.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:30.806+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:31.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:30.806+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:31.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:31.524+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:31.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:31.524+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:31.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:31.524+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:31.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:31.525+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:32.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:31.723+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:32.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:31.723+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:32.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:31.723+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:32.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:31.724+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:32.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:31.828+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:32.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:31.828+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:32.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:31.828+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:32.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:31.829+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:32.813 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:32.510+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:32.813 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:32.510+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:32.813 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:32.510+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:32.813 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:32.511+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:32.813 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:32.747+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:32.814 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:32.747+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:32.814 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:32.747+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:32.814 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:32.748+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:33.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:32.812+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:33.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:32.812+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:33.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:32.812+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:33.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:32.813+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:33.793 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:33.538+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:33.793 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:33.538+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:33.793 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:33.538+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:33.793 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:33.539+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:33.794 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:33.774+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:33.794 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:33.774+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:33.794 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:33.774+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:33.794 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:33.775+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:34.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:33.792+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:34.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:33.792+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:34.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:33.792+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:34.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:33.792+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:34.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:34.580+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:34.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:34.580+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:34.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:34.580+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:34.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:34.580+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:35.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:34.754+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:35.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:34.754+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:35.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:34.754+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:35.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:34.754+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:35.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:34.832+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:35.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:34.832+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:35.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:34.832+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:35.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:34.833+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:35.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:21:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:21:35.562+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27137 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:21:35.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:35.781+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:35.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:35.781+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:35.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:35.781+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:35.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:35.781+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:35.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:35.787+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:35.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:35.787+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:35.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:35.787+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:35.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:35.788+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:35.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:35.606+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:35.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:35.606+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:35.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:35.606+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:35.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:35.607+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:36.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:36.583+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:36.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:36.583+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:36.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:36.583+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:36.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:36.584+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:36.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:36.732+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:36.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:36.732+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:36.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:36.732+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:36.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:36.733+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:36.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:36.820+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:36.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:36.820+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:36.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:36.820+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:36.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:36.821+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:37.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:37.838+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:37.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:37.838+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:37.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:37.838+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:37.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:37.839+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:37.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:37.630+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:37.977 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:37.630+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:37.977 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:37.630+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:37.977 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:37.630+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:37.977 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:37.769+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:37.977 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:37.770+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:37.977 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:37.770+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:37.977 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:37.770+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:38.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:38.592+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:38.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:38.592+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:38.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:38.592+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:38.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:38.593+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:38.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:38.805+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:38.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:38.805+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:38.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:38.805+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:38.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:38.806+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:38.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:38.736+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:38.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:38.736+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:38.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:38.736+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:38.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:38.737+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:39.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:39.750+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:39.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:39.750+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:39.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:39.750+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:39.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:39.751+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:39.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:39.829+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:39.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:39.829+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:39.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:39.829+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:39.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:39.830+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:39.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:39.616+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:39.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:39.616+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:39.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:39.616+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:39.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:39.616+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:40.820 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:21:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:21:40.565+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27220 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:21:40.820 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:40.712+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:40.820 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:40.712+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:40.821 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:40.712+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:40.821 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:40.712+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:40.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:40.599+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:40.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:40.599+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:40.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:40.599+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:40.821 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:40.599+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:41.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:40.819+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:41.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:40.819+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:41.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:40.819+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:41.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:40.819+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:41.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:41.561+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:41.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:41.561+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:41.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:41.561+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:41.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:41.561+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:41.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:41.788+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:41.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:41.788+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:41.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:41.788+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:41.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:41.788+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:41.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:41.752+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:41.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:41.752+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:41.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:41.752+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:41.977 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:41.752+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:42.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:42.735+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:42.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:42.735+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:42.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:42.735+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:42.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:42.736+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:42.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:42.544+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:42.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:42.544+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:42.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:42.544+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:42.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:42.545+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:42.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:42.748+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:42.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:42.748+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:42.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:42.748+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:42.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:42.748+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:43.773 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:43.512+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:43.773 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:43.512+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:43.774 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:43.512+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:43.774 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:43.513+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:44.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:43.787+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:44.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:43.787+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:44.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:43.787+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:44.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:43.788+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:44.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:43.773+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:44.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:43.773+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:44.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:43.773+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:44.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:43.774+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:44.763 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:44.500+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:44.763 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:44.500+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:44.763 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:44.500+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:44.764 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:44.501+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:44.764 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:44.739+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:44.765 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:44.739+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:44.765 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:44.739+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:44.765 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:44.740+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:45.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:44.762+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:45.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:44.762+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:45.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:44.762+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:45.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:44.763+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:45.770 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:21:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:21:45.569+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27305 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:21:45.771 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:45.474+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:45.771 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:45.474+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:45.771 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:45.474+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:45.771 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:45.475+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:45.772 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:45.699+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:45.772 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:45.699+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:45.772 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:45.699+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:45.772 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:45.700+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:46.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:45.770+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:46.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:45.770+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:46.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:45.770+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:46.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:45.771+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:46.686 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:46.686+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:46.687 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:46.686+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:46.687 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:46.429+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:46.687 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:46.429+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:46.687 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:46.429+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:46.687 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:46.430+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:46.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:46.686+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:46.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:46.687+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:46.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:46.765+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:46.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:46.765+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:46.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:46.765+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:46.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:46.766+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:47.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:47.466+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:47.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:47.466+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:47.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:47.466+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:47.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:47.467+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:47.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:47.681+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:47.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:47.681+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:47.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:47.681+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:47.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:47.682+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:48.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:47.750+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:48.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:47.750+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:48.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:47.750+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:48.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:47.750+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:48.710 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:48.448+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:48.710 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:48.448+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:48.710 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:48.448+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:48.710 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:48.449+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:48.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:48.716+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:48.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:48.716+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:48.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:48.716+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:48.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:48.717+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:48.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:48.710+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:48.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:48.710+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:48.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:48.710+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:48.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:48.711+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:49.720 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:49.435+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:49.720 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:49.435+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:49.720 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:49.435+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:49.720 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:49.435+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:49.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:49.756+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:49.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:49.756+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:49.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:49.756+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:49.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:49.757+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:49.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:49.720+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:49.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:49.720+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:49.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:49.720+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:49.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:49.721+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:50.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:50.450+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:50.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:50.450+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:50.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:50.450+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:50.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:50.451+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:50.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:21:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:21:50.572+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27388 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:21:50.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:50.747+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:50.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:50.747+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:50.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:50.747+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:50.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:50.748+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:50.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:50.777+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:50.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:50.777+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:50.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:50.777+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:50.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:50.778+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:51.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:51.428+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:51.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:51.428+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:51.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:51.428+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:51.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:51.429+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:52.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:51.743+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:52.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:51.743+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:52.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:51.743+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:52.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:51.743+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:52.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:51.740+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:52.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:51.740+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:52.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:51.740+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:52.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:51.741+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:52.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:52.430+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:52.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:52.430+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:52.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:52.430+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:52.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:52.431+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:53.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:52.731+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:53.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:52.731+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:53.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:52.731+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:53.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:52.732+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:53.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:52.763+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:53.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:52.763+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:53.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:52.763+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:53.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:52.764+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:53.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:53.404+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:53.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:53.404+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:53.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:53.404+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:53.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:53.405+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:53.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:53.730+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:53.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:53.730+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:53.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:53.730+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:53.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:53.730+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:53.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:53.701+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:53.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:53.701+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:53.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:53.701+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:53.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:53.703+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:54.714 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:54.381+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:54.714 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:54.381+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:54.714 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:54.381+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:54.714 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:54.382+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:54.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:54.721+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:54.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:54.721+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:54.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:54.721+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:54.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:54.722+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:54.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:54.714+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:54.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:54.714+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:54.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:54.714+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:54.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:54.715+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:55.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:21:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:21:55.575+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27474 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:21:55.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:55.373+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:55.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:55.373+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:55.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:55.373+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:55.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:55.374+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:55.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:55.760+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:55.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:55.760+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:55.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:55.760+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:55.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:55.761+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:55.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:55.726+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:55.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:55.726+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:55.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:55.726+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:55.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:55.727+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:56.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:56.404+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:56.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:56.404+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:56.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:56.404+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:56.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:56.405+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:57.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:56.728+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:57.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:56.728+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:57.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:56.728+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:57.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:56.729+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:57.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:56.768+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:57.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:56.768+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:57.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:56.768+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:57.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:56.768+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:57.718 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:57.454+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:57.718 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:57.454+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:57.718 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:57.454+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:57.719 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:57.455+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:57.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:57.718+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:57.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:57.718+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:57.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:57.718+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:57.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:57.719+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:57.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:57.790+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:57.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:57.790+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:57.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:57.790+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:57.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:57.790+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:58.719 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:58.411+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:58.719 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:58.411+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:58.719 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:58.411+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:58.720 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:58.412+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:58.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:58.807+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:58.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:58.807+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:58.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:58.807+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:58.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:58.808+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:58.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:58.719+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:58.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:58.719+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:58.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:58.719+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:58.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:58.719+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:59.713 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:59.436+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:59.714 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:59.436+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:59.714 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:59.436+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:21:59.714 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:21:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:21:59.437+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:21:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:59.828+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:59.828+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:59.828+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:21:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:21:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:21:59.828+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:21:59.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:59.713+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:59.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:59.713+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:59.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:59.713+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:21:59.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:21:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:21:59.714+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:00.821 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:22:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:22:00.578+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27557 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:22:00.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:00.484+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:00.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:00.484+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:00.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:00.484+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:00.822 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:00.485+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:00.823 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:00.720+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:00.823 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:00.720+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:00.823 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:00.720+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:00.823 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:00.721+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:01.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:00.821+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:01.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:00.821+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:01.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:00.821+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:01.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:00.823+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:01.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:01.451+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:01.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:01.451+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:01.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:01.451+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:01.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:01.452+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:02.118 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:01.861+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:02.118 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:01.861+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:02.118 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:01.861+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:02.118 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:01.861+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:02.119 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:01.765+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:02.119 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:01.765+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:02.119 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:01.765+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:02.119 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:01.765+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:02.768 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:02.487+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:02.768 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:02.487+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:02.768 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:02.487+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:02.768 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:02.487+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:03.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:02.814+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:03.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:02.814+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:03.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:02.814+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:03.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:02.815+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:03.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:02.768+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:03.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:02.768+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:03.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:02.768+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:03.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:02.769+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:03.856 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:03.528+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:03.857 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:03.528+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:03.857 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:03.528+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:03.857 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:03.529+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:03.857 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:03.756+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:03.857 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:03.756+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:03.858 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:03.756+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:03.858 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:03.757+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:04.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:03.856+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:04.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:03.856+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:04.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:03.856+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:04.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:03.857+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:04.896 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:04.524+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:04.896 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:04.524+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:04.896 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:04.524+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:04.896 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:04.525+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:04.897 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:04.768+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:04.897 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:04.768+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:04.897 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:04.768+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:04.897 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:04.769+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:05.190 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:04.896+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:05.190 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:04.896+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:05.190 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:04.896+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:05.191 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:04.897+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:05.745 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:22:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:22:05.581+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27643 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:22:05.745 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:05.481+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:05.746 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:05.481+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:05.746 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:05.481+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:05.746 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:05.482+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:06.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:05.891+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:06.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:05.891+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:06.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:05.891+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:06.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:05.892+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:06.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:05.745+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:06.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:05.745+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:06.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:05.745+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:06.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:05.746+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:06.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:06.438+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:06.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:06.438+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:06.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:06.438+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:06.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:06.439+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:07.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:06.941+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:07.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:06.941+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:07.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:06.941+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:07.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:06.942+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:07.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:06.772+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:07.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:06.772+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:07.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:06.772+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:07.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:06.772+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:07.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:07.401+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:07.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:07.401+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:07.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:07.401+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:07.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:07.402+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:08.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:07.978+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:08.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:07.978+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:08.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:07.978+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:08.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:07.979+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:08.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:07.762+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:08.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:07.762+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:08.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:07.762+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:08.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:07.763+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:08.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:08.398+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:08.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:08.398+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:08.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:08.398+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:08.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:08.399+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:09.023 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:08.760+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:09.023 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:08.760+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:09.023 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:08.760+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:09.023 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:08.761+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:09.410 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:09.023+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:09.410 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:09.023+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:09.410 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:09.023+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:09.410 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:09.024+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:09.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:09.410+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:09.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:09.410+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:09.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:09.410+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:09.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:09.411+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:10.206 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:10.012+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:10.206 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:10.012+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:10.206 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:10.012+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:10.206 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:10.013+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:10.207 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:09.764+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:10.207 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:09.764+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:10.207 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:09.764+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:10.207 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:09.765+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:10.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:10.451+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:10.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:10.451+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:10.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:10.451+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:10.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:10.452+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:10.965 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:22:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:22:10.584+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27726 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:22:10.965 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:10.729+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:10.965 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:10.729+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:10.965 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:10.729+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:10.966 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:10.730+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:11.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:10.964+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:11.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:10.964+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:11.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:10.964+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:11.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:10.965+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:11.683 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:11.484+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:11.683 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:11.484+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:11.683 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:11.484+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:11.683 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:11.485+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:11.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:11.682+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:11.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:11.682+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:11.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:11.682+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:11.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:11.683+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:12.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:11.982+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:12.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:11.982+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:12.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:11.982+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:12.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:11.983+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:12.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:12.469+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:12.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:12.469+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:12.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:12.469+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:12.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:12.470+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:12.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:12.648+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:12.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:12.648+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:12.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:12.648+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:12.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:12.649+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:13.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:13.018+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:13.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:13.018+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:13.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:13.018+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:13.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:13.018+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:13.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:13.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:13.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:13.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:13.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:13.626+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:13.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:13.627+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:13.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:13.505+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:13.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:13.505+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:13.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:13.505+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:13.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:13.506+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:14.406 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:14.015+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:14.406 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:14.015+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:14.406 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:14.015+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:14.406 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:14.016+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:14.676 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:14.484+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:14.676 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:14.484+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:14.676 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:14.484+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:14.676 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:14.484+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:14.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:14.676+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:14.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:14.676+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:14.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:14.676+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:14.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:14.677+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:15.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:15.058+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:15.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:15.058+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:15.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:15.058+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:15.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:15.058+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:15.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:22:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:22:15.587+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27812 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:22:15.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:15.695+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:15.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:15.695+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:15.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:15.695+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:15.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:15.696+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:15.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:15.533+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:15.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:15.533+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:15.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:15.533+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:15.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:15.534+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:16.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:16.016+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:16.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:16.016+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:16.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:16.016+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:16.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:16.017+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:16.865 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:16.511+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:16.865 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:16.511+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:16.865 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:16.511+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:16.865 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:16.511+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:16.866 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:16.660+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:16.866 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:16.660+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:16.866 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:16.660+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:16.866 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:16.660+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:17.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:17.036+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:17.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:17.036+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:17.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:17.036+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:17.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:17.037+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:17.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:17.480+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:17.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:17.480+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:17.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:17.480+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:17.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:17.481+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:17.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:17.690+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:17.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:17.690+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:17.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:17.690+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:17.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:17.691+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:18.439 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:18.004+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:18.439 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:18.004+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:18.439 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:18.004+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:18.439 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:18.005+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:18.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:18.439+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:18.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:18.439+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:18.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:18.439+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:18.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:18.440+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:18.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:18.700+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:18.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:18.700+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:18.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:18.700+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:18.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:18.700+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:19.426 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:19.004+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:19.426 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:19.004+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:19.426 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:19.004+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:19.426 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:19.005+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:19.677 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:19.426+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:19.677 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:19.426+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:19.677 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:19.426+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:19.677 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:19.427+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:19.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:19.677+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:19.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:19.677+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:19.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:19.677+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:19.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:19.678+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:20.377 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:19.980+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:20.378 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:19.980+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:20.378 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:19.980+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:20.378 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:19.980+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:20.647 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:22:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:22:20.591+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27895 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:22:20.648 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:20.377+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:20.648 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:20.377+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:20.648 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:20.377+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:20.648 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:20.378+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:20.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:20.647+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:20.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:20.647+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:20.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:20.647+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:20.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:20.648+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:21.345 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:20.989+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:21.345 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:20.989+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:21.345 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:20.989+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:21.345 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:20.990+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:21.654 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:21.345+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:21.655 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:21.345+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:21.655 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:21.345+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:21.655 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:21.345+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:21.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:21.654+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:21.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:21.654+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:21.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:21.654+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:21.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:21.655+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:22.394 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:21.976+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:22.394 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:21.976+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:22.394 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:21.976+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:22.394 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:21.977+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:22.648 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:22.394+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:22.649 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:22.394+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:22.649 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:22.394+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:22.649 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:22.394+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:22.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:22.648+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:22.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:22.648+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:22.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:22.648+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:22.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:22.649+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:23.376 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:23.015+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:23.376 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:23.015+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:23.376 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:23.015+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:23.377 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:23.015+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:23.693 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:23.376+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:23.693 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:23.376+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:23.693 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:23.376+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:23.694 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:23.377+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:23.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:23.693+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:23.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:23.693+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:23.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:23.693+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:23.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:23.694+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:24.382 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:24.029+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:24.382 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:24.029+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:24.382 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:24.029+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:24.382 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:24.030+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:24.672 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:24.382+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:24.672 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:24.382+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:24.672 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:24.382+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:24.672 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:24.382+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:24.973 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:24.671+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:24.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:24.672+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:24.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:24.672+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:24.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:24.672+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:25.346 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:25.025+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:25.346 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:25.025+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:25.346 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:25.025+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:25.347 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:25.026+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:25.605 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:22:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:22:25.594+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27981 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:22:25.605 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:25.346+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:25.605 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:25.346+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:25.605 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:25.346+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:25.606 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:25.347+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:25.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:25.700+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:25.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:25.700+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:25.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:25.700+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:25.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:25.701+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:26.337 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:25.996+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:26.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:25.996+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:26.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:25.996+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:26.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:25.997+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:26.710 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:26.337+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:26.710 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:26.337+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:26.710 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:26.337+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:26.710 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:26.338+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:26.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:26.709+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:26.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:26.709+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:26.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:26.709+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:26.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:26.710+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:27.322 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:26.984+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:27.323 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:26.984+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:27.323 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:26.984+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:27.323 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:26.985+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:27.709 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:27.322+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:27.710 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:27.322+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:27.710 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:27.322+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:27.710 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:27.323+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:27.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:27.709+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:27.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:27.709+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:27.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:27.709+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:27.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:27.710+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:28.253 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:28.032+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:28.253 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:28.032+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:28.253 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:28.032+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:28.253 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:28.033+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:28.680 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:28.290+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:28.681 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:28.290+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:28.681 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:28.290+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:28.681 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:28.291+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:28.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:28.680+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:28.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:28.680+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:28.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:28.680+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:28.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:28.681+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:29.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:29.300+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:29.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:29.300+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:29.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:29.300+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:29.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:29.301+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:29.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:29.064+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:29.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:29.064+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:29.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:29.064+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:29.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:29.065+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:29.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:29.654+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:29.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:29.654+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:29.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:29.654+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:29.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:29.655+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:30.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:30.300+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:30.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:30.300+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:30.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:30.300+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:30.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:30.301+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:30.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:30.068+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:30.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:30.068+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:30.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:30.068+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:30.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:30.069+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:30.973 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:22:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:22:30.597+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28064 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:22:30.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:30.676+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:30.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:30.676+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:30.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:30.676+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:30.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:30.677+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:31.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:31.324+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:31.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:31.324+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:31.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:31.324+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:31.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:31.324+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:31.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:31.084+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:31.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:31.084+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:31.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:31.084+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:31.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:31.085+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:31.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:31.691+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:31.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:31.691+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:31.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:31.691+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:31.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:31.692+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:32.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:32.104+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:32.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:32.104+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:32.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:32.104+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:32.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:32.105+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:32.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:32.334+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:32.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:32.334+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:32.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:32.334+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:32.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:32.334+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:32.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:32.675+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:32.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:32.675+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:32.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:32.675+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:32.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:32.675+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:33.378 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:33.070+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:33.378 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:33.070+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:33.378 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:33.070+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:33.378 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:33.070+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:33.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:33.378+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:33.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:33.378+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:33.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:33.378+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:33.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:33.379+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:33.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:33.699+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:33.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:33.699+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:33.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:33.699+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:33.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:33.700+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:34.371 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:34.092+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:34.371 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:34.092+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:34.371 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:34.092+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:34.371 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:34.092+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:34.684 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:34.370+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:34.685 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:34.371+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:34.685 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:34.371+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:34.686 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:34.372+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:34.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:34.684+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:34.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:34.684+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:34.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:34.684+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:34.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:34.685+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:35.360 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:35.078+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:35.360 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:35.078+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:35.360 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:35.078+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:35.360 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:35.079+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:35.611 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:22:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:22:35.600+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28149 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:22:35.611 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:35.360+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:35.611 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:35.360+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:35.612 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:35.360+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:35.612 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:35.361+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:35.910 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:35.720+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:35.910 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:35.720+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:35.911 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:35.720+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:35.911 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:35.721+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:36.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:36.060+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:36.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:36.060+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:36.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:36.060+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:36.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:36.061+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:36.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:36.383+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:36.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:36.383+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:36.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:36.383+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:36.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:36.384+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:36.761+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:36.761+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:36.761+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:37.089 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:36.762+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:37.424 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:37.089+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:37.424 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:37.089+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:37.424 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:37.089+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:37.424 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:37.090+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:37.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:37.424+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:37.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:37.424+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:37.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:37.424+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:37.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:37.424+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:38.051 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:37.762+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:38.051 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:37.762+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:38.051 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:37.762+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:38.051 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:37.762+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:38.453 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:38.050+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:38.453 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:38.050+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:38.453 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:38.050+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:38.453 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:38.051+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:38.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:38.452+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:38.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:38.452+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:38.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:38.452+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:38.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:38.453+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:39.033 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:38.755+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:39.033 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:38.755+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:39.033 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:38.755+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:39.033 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:38.756+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:39.467 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:39.033+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:39.467 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:39.033+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:39.468 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:39.033+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:39.468 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:39.034+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:39.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:39.467+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:39.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:39.467+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:39.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:39.467+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:39.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:39.468+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:40.044 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:39.779+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:40.044 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:39.779+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:40.044 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:39.779+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:40.044 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:39.779+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:40.357 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:40.044+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:40.357 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:40.044+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:40.357 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:40.044+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:40.357 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:40.044+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:40.613 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:22:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:22:40.603+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28232 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:22:40.613 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:40.513+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:40.613 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:40.513+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:40.613 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:40.513+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:40.614 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:40.514+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:40.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:40.794+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:40.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:40.794+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:40.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:40.794+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:40.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:40.794+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:41.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:41.031+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:41.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:41.031+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:41.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:41.031+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:41.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:41.032+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:41.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:41.791+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:41.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:41.791+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:41.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:41.791+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:41.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:41.792+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:41.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:41.550+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:41.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:41.550+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:41.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:41.550+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:41.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:41.551+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:42.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:42.026+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:42.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:42.026+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:42.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:42.026+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:42.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:42.027+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:42.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:42.548+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:42.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:42.548+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:42.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:42.548+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:42.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:42.548+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:42.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:42.784+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:42.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:42.784+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:42.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:42.784+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:42.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:42.785+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:43.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:43.048+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:43.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:43.048+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:43.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:43.048+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:43.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:43.049+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:43.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:43.527+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:43.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:43.527+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:43.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:43.527+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:43.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:43.528+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:43.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:43.764+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:43.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:43.764+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:43.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:43.764+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:43.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:43.764+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:44.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:44.086+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:44.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:44.086+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:44.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:44.086+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:44.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:44.087+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:44.778 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:44.513+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:44.779 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:44.513+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:44.779 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:44.513+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:44.779 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:44.514+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:45.126 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:44.777+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:45.126 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:44.777+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:45.126 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:44.777+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:45.126 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:44.778+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:45.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:45.131+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:45.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:45.131+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:45.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:45.131+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:45.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:45.131+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:45.796 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:22:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:22:45.605+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28318 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:22:45.796 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:45.540+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:45.796 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:45.540+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:45.796 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:45.540+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:45.797 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:45.540+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:46.132 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:45.794+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:46.132 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:45.794+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:46.132 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:45.794+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:46.132 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:45.795+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:46.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:46.130+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:46.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:46.131+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:46.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:46.131+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:46.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:46.131+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:46.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:46.751+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:46.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:46.751+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:46.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:46.752+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:46.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:46.752+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:46.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:46.563+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:46.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:46.563+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:46.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:46.563+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:46.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:46.564+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:47.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:47.168+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:47.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:47.168+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:47.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:47.168+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:47.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:47.169+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:47.772 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:47.577+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:47.772 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:47.577+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:47.772 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:47.577+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:47.772 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:47.578+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:48.136 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:47.770+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:48.137 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:47.770+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:48.137 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:47.770+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:48.137 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:47.771+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:48.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:48.135+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:48.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:48.135+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:48.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:48.135+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:48.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:48.136+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:48.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:48.733+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:48.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:48.733+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:48.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:48.733+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:48.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:48.734+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:48.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:48.593+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:48.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:48.593+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:48.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:48.593+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:48.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:48.594+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:49.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:49.110+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:49.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:49.110+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:49.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:49.110+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:49.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:49.110+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:49.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:49.593+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:49.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:49.593+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:49.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:49.593+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:49.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:49.594+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:49.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:49.767+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:49.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:49.767+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:49.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:49.767+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:49.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:49.767+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:50.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:50.068+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:50.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:50.068+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:50.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:50.068+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:50.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:50.068+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:50.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:22:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:22:50.608+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28401 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:22:50.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:50.809+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:50.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:50.809+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:50.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:50.809+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:50.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:50.810+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:50.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:50.596+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:50.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:50.596+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:50.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:50.596+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:50.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:50.597+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:51.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:51.022+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:51.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:51.022+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:51.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:51.022+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:51.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:51.022+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:51.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:51.615+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:51.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:51.615+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:51.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:51.615+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:51.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:51.616+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:51.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:51.801+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:51.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:51.801+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:51.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:51.801+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:51.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:51.801+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:52.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:52.004+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:52.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:52.004+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:52.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:52.004+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:52.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:52.005+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:52.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:52.623+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:52.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:52.623+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:52.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:52.623+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:52.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:52.624+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:52.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:52.753+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:52.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:52.753+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:52.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:52.753+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:52.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:52.753+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:53.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:53.020+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:53.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:53.020+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:53.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:53.020+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:53.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:53.021+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:53.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:53.600+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:53.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:53.600+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:53.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:53.600+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:53.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:53.601+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:53.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:53.767+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:53.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:53.767+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:53.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:53.767+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:53.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:53.767+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:54.342 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:53.999+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:54.342 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:53.999+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:54.342 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:53.999+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:54.342 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:54.000+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:54.958 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:54.601+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:54.958 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:54.601+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:54.958 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:54.601+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:54.959 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:54.602+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:54.959 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:54.788+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:54.959 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:54.788+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:54.959 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:54.788+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:54.959 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:54.789+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:55.223 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:54.957+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:55.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:54.957+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:55.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:54.957+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:55.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:54.958+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:55.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:22:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:22:55.611+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28487 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:22:55.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:55.753+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:55.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:55.753+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:55.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:55.753+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:55.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:55.754+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:55.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:55.607+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:55.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:55.607+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:55.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:55.607+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:55.977 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:55.608+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:56.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:55.996+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:56.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:55.996+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:56.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:55.996+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:56.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:55.996+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:56.963 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:56.644+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:56.963 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:56.644+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:56.963 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:56.644+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:56.963 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:56.645+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:56.964 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:56.739+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:56.964 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:56.739+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:56.964 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:56.739+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:56.964 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:56.740+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:57.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:56.961+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:57.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:56.961+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:57.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:56.961+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:57.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:56.962+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:57.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:57.610+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:57.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:57.610+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:57.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:57.610+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:57.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:57.611+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:57.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:57.764+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:57.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:57.764+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:57.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:57.764+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:57.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:57.765+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:58.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:57.978+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:58.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:57.978+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:58.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:57.978+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:58.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:57.979+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:58.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:58.612+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:58.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:58.613+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:58.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:58.613+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:58.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:58.614+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:58.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:58.740+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:58.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:58.740+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:58.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:58.740+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:58.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:58.741+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:59.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:58.945+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:59.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:58.945+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:59.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:58.945+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:22:59.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:58.946+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:22:59.925 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:59.586+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:59.925 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:59.586+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:59.925 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:59.586+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:22:59.925 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:22:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:22:59.587+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:22:59.925 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:59.742+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:59.926 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:59.742+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:59.926 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:59.742+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:22:59.926 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:22:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:22:59.743+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:00.209 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:59.924+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:00.209 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:59.924+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:00.210 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:59.924+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:00.210 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:22:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:22:59.925+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:00.739 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:23:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:23:00.614+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28570 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:23:00.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:00.587+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:00.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:00.587+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:00.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:00.587+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:00.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:00.587+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:01.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:00.897+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:01.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:00.897+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:01.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:00.897+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:01.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:00.897+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:01.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:00.758+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:01.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:00.758+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:01.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:00.758+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:01.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:00.759+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:01.940 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:01.620+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:01.941 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:01.620+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:01.941 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:01.620+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:01.941 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:01.620+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:01.941 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:01.746+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:01.941 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:01.746+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:01.941 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:01.746+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:01.941 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:01.746+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:02.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:01.940+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:02.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:01.940+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:02.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:01.940+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:02.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:01.940+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:02.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:02.616+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:02.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:02.616+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:02.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:02.616+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:02.945 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:02.616+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:02.945 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:02.706+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:02.945 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:02.706+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:02.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:02.706+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:02.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:02.706+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:03.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:02.943+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:03.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:02.943+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:03.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:02.943+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:03.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:02.944+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:03.938 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:03.615+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:03.939 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:03.615+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:03.939 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:03.615+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:03.939 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:03.615+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:03.939 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:03.726+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:03.939 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:03.726+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:03.939 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:03.726+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:03.939 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:03.727+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:04.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:03.938+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:04.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:03.938+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:04.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:03.938+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:04.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:03.939+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:04.940 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:04.604+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:04.940 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:04.604+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:04.940 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:04.604+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:04.940 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:04.605+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:04.941 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:04.748+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:04.941 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:04.748+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:04.941 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:04.748+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:04.941 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:04.749+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:05.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:04.940+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:05.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:04.940+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:05.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:04.940+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:05.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:04.941+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:05.898 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:23:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:23:05.618+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28656 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:23:05.898 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:05.601+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:05.898 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:05.601+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:05.898 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:05.601+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:05.899 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:05.602+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:05.899 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:05.764+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:05.899 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:05.764+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:05.899 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:05.764+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:05.899 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:05.765+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:06.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:05.898+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:06.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:05.898+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:06.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:05.898+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:06.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:05.899+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:06.877 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:06.582+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:06.877 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:06.582+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:06.877 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:06.582+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:06.877 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:06.583+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:06.878 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:06.794+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:06.878 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:06.794+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:06.878 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:06.794+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:06.878 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:06.795+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:07.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:06.877+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:07.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:06.877+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:07.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:06.877+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:07.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:06.877+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:07.845 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:07.565+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:07.845 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:07.565+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:07.845 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:07.565+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:07.845 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:07.565+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:07.845 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:07.758+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:07.845 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:07.758+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:07.846 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:07.758+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:07.846 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:07.759+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:08.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:07.844+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:08.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:07.844+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:08.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:07.844+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:08.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:07.845+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:08.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:08.559+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:08.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:08.559+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:08.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:08.559+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:08.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:08.559+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:08.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:08.805+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:08.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:08.805+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:08.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:08.805+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:08.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:08.806+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:08.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:08.780+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:08.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:08.780+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:08.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:08.780+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:08.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:08.780+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:09.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:09.599+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:09.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:09.599+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:09.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:09.599+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:09.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:09.601+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:09.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:09.787+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:09.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:09.787+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:09.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:09.787+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:09.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:09.787+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:09.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:09.788+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:09.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:09.788+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:09.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:09.788+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:09.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:09.789+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:10.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:23:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:23:10.621+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28739 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:23:10.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:10.586+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:10.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:10.586+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:10.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:10.586+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:10.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:10.587+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:10.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:10.830+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:10.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:10.830+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:10.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:10.830+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:10.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:10.831+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:10.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:10.780+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:10.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:10.780+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:10.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:10.780+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:10.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:10.781+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:11.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:11.632+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:11.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:11.632+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:11.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:11.632+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:11.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:11.633+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:11.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:11.830+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:11.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:11.830+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:11.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:11.830+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:11.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:11.831+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:11.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:11.815+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:11.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:11.815+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:11.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:11.815+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:11.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:11.815+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:12.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:12.771+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:12.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:12.771+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:12.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:12.771+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:12.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:12.772+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:12.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:12.618+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:12.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:12.618+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:12.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:12.618+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:12.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:12.619+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:12.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:12.788+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:12.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:12.788+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:12.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:12.788+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:12.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:12.789+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:13.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:13.586+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:13.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:13.586+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:13.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:13.586+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:13.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:13.587+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:13.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:13.802+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:13.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:13.802+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:13.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:13.802+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:13.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:13.803+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:13.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:13.783+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:13.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:13.783+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:13.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:13.783+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:13.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:13.783+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:14.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:14.564+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:14.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:14.564+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:14.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:14.564+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:14.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:14.565+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:14.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:14.780+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:14.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:14.780+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:14.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:14.780+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:14.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:14.781+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:14.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:14.758+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:14.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:14.758+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:14.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:14.758+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:14.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:14.759+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:15.809 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:23:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:23:15.625+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28825 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:23:15.810 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:15.544+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:15.810 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:15.544+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:15.810 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:15.544+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:15.810 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:15.545+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:15.811 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:15.736+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:15.811 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:15.736+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:15.811 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:15.736+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:15.811 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:15.737+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:16.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:15.809+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:16.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:15.809+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:16.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:15.809+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:16.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:15.810+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:16.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:16.737+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:16.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:16.737+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:16.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:16.737+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:16.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:16.738+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:16.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:16.561+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:16.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:16.561+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:16.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:16.561+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:16.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:16.562+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:16.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:16.801+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:16.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:16.801+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:16.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:16.801+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:16.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:16.801+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:17.876 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:17.612+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:17.876 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:17.612+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:17.876 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:17.612+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:17.877 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:17.612+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:17.877 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:17.747+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:17.877 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:17.747+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:17.877 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:17.747+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:17.877 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:17.748+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:17.878 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:17.820+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:17.878 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:17.820+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:17.878 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:17.820+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:17.878 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:17.820+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:18.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:18.742+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:18.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:18.742+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:18.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:18.742+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:18.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:18.743+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:18.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:18.649+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:18.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:18.649+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:18.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:18.649+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:18.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:18.650+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:18.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:18.845+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:18.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:18.845+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:18.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:18.845+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:18.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:18.846+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:19.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:19.766+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:19.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:19.766+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:19.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:19.766+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:19.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:19.767+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:19.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:19.642+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:19.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:19.642+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:19.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:19.642+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:19.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:19.642+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:19.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:19.806+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:19.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:19.806+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:19.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:19.806+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:19.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:19.806+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:20.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:23:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:23:20.628+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28908 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:23:20.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:20.608+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:20.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:20.608+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:20.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:20.608+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:20.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:20.608+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:20.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:20.838+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:20.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:20.838+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:20.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:20.838+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:20.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:20.839+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:20.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:20.719+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:20.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:20.719+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:20.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:20.719+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:20.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:20.719+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:21.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:21.627+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:21.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:21.627+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:21.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:21.627+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:21.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:21.628+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:21.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:21.752+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:21.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:21.752+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:21.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:21.752+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:21.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:21.753+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:21.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:21.809+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:21.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:21.809+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:21.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:21.809+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:21.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:21.810+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:22.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:22.614+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:22.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:22.614+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:22.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:22.614+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:22.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:22.615+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:22.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:22.764+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:22.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:22.764+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:22.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:22.764+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:22.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:22.765+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:22.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:22.855+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:22.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:22.855+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:22.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:22.855+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:22.976 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:22.856+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:23.877 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:23.726+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:23.877 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:23.726+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:23.877 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:23.726+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:23.877 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:23.727+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:23.878 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:23.589+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:23.878 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:23.589+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:23.878 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:23.589+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:23.878 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:23.589+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:24.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:23.877+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:24.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:23.877+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:24.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:23.877+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:24.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:23.877+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:24.860 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:24.605+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:24.861 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:24.605+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:24.861 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:24.605+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:24.861 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:24.606+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:24.861 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:24.749+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:24.861 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:24.749+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:24.862 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:24.749+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:24.862 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:24.750+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:25.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:24.860+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:25.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:24.860+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:25.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:24.860+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:25.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:24.861+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:25.899 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:23:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:23:25.631+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28989 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:23:25.900 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:25.632+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:25.900 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:25.632+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:25.900 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:25.633+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:25.900 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:25.634+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:25.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:25.778+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:25.901 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:25.778+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:25.901 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:25.778+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:25.901 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:25.779+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:26.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:25.899+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:26.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:25.899+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:26.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:25.899+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:26.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:25.900+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:26.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:26.658+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:26.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:26.658+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:26.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:26.658+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:26.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:26.659+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:26.945 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:26.761+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:26.945 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:26.761+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:26.945 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:26.761+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:26.945 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:26.761+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:27.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:26.944+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:27.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:26.944+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:27.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:26.944+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:27.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:26.944+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:27.961 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:27.611+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:27.962 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:27.612+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:27.962 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:27.612+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:27.962 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:27.613+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:27.962 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:27.791+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:27.962 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:27.791+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:27.962 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:27.791+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:27.962 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:27.792+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:28.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:27.961+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:28.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:27.961+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:28.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:27.961+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:12:55.121796+0000 (oldest deadline 2024-01-23T09:13:15.121796+0000) 2024-01-23T09:23:28.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:27.962+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:28.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:28.645+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:28.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:28.645+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:28.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:28.645+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:28.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:28.646+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:28.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:28.828+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:28.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:28.828+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:28.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:28.828+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:28.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:28.829+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:29.245 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:28.977+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:29.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:29.665+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:29.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:29.665+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:29.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:29.665+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:29.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:29.666+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:29.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:29.810+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:29.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:29.810+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:29.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:29.810+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:29.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:29.811+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:30.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:30.025+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:30.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:23:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:23:30.634+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29072 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:23:30.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:30.663+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:30.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:30.663+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:30.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:30.663+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:30.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:30.663+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:30.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:30.829+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:30.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:30.829+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:30.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:30.829+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:30.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:30.830+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:31.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:31.013+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:31.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:31.700+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:31.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:31.700+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:31.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:31.700+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:31.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:31.701+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:31.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:31.851+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:31.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:31.851+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:31.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:31.851+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:31.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:31.852+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:32.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:32.018+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:32.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:32.656+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:32.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:32.656+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:32.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:32.656+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:32.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:32.657+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:32.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:32.804+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:32.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:32.805+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:32.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:32.805+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:32.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:32.805+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:33.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:33.051+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:33.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:33.674+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:33.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:33.674+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:33.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:33.674+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:33.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:33.675+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:33.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:33.803+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:33.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:33.803+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:33.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:33.803+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:33.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:33.804+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:34.278 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:34.023+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:34.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:34.722+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:34.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:34.722+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:34.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:34.722+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:34.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:34.723+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:34.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:34.806+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:34.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:34.806+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:34.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:34.806+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:34.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:34.807+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:35.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:34.987+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:35.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:23:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:23:35.638+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29158 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:23:35.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:35.698+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:35.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:35.698+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:35.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:35.698+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:35.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:35.698+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:35.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:35.781+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:35.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:35.781+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:35.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:35.781+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:35.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:35.781+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:36.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:36.028+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:36.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:36.691+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:36.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:36.691+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:36.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:36.691+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:03.209424+0000 (oldest deadline 2024-01-23T09:13:23.209424+0000) 2024-01-23T09:23:36.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:36.692+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:36.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:36.804+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:36.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:36.804+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:36.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:36.804+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:36.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:36.805+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:37.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:37.000+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:37.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:37.685+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:37.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:37.769+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:37.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:37.769+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:37.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:37.769+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:37.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:37.770+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:38.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:38.016+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:38.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:38.645+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:38.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:38.795+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:38.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:38.796+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:38.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:38.796+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:38.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:38.797+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:39.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:39.052+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:39.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:39.685+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:39.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:39.830+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:39.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:39.830+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:39.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:39.830+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:39.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:39.831+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:40.422 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:40.088+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:40.724 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:23:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:23:40.641+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29241 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:23:40.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:40.654+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:41.076 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:40.827+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:41.076 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:40.827+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:41.077 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:40.827+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:41.077 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:40.828+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:41.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:41.076+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:41.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:41.700+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:41.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:41.834+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:41.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:41.834+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:41.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:41.834+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:41.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:41.835+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:42.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:42.079+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:42.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:42.708+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:42.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:42.813+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:42.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:42.813+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:42.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:42.813+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:42.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:42.814+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:43.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:43.037+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:43.773 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:43.742+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:44.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:44.008+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:44.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:43.772+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:44.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:43.772+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:44.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:43.772+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:44.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:43.774+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:45.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:44.963+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:45.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:44.757+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:45.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:44.753+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:45.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:44.753+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:45.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:44.753+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:45.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:44.754+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:45.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:23:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:23:45.644+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29327 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:23:45.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:45.725+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:45.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:45.773+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:45.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:45.773+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:45.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:45.773+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:45.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:45.774+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:46.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:46.014+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:46.971 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:46.756+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:46.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:46.785+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:46.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:46.785+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:46.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:46.785+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:13:13.977894+0000 (oldest deadline 2024-01-23T09:13:33.977894+0000) 2024-01-23T09:23:46.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:46.786+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:47.223 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:46.971+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:48.125 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:47.753+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:48.125 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:47.822+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:48.126 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:47.947+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:49.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:48.760+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:49.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:48.854+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:49.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:48.925+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:50.211 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:49.743+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:50.212 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:49.889+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:50.212 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:49.973+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:50.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:50.857+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:50.951 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:23:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:23:50.648+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29410 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:23:50.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:50.757+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:51.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:50.950+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:52.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:51.777+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:52.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:51.848+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:52.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:51.945+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:52.965 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:52.886+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:52.966 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:52.735+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:53.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:52.965+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:53.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:53.704+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:54.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:53.922+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:54.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:53.919+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:23:54.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:53.919+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:23:54.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:53.919+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:23:54.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:53.920+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:55.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:54.969+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:55.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:54.734+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:55.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:54.934+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:23:55.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:54.934+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:23:55.226 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:54.934+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:23:55.226 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:54.935+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:55.956 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:23:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:23:55.651+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29496 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:23:55.956 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:55.696+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:56.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:56.002+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:56.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:55.956+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:23:56.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:55.956+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:23:56.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:55.956+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:23:56.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:55.957+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:56.958 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:56.658+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:57.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:56.973+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:57.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:56.958+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:23:57.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:56.958+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:23:57.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:56.958+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:23:57.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:56.958+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:57.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:57.665+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:23:57.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:57.665+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:23:57.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:57.665+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:23:57.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:57.666+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:58.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:58.003+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:58.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:57.980+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:23:58.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:57.980+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:23:58.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:57.980+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:23:58.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:57.981+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:58.958 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:58.626+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:23:58.958 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:58.626+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:23:58.958 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:58.626+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:23:58.958 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:58.627+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:59.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:58.958+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:59.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:58.982+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:23:59.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:58.982+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:23:59.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:58.982+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:23:59.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:58.982+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:23:59.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:23:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:23:59.913+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:23:59.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:59.668+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:23:59.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:59.668+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:23:59.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:59.668+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:23:59.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:23:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:23:59.669+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:00.211 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:59.950+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:00.212 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:59.950+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:00.212 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:59.950+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:00.212 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:23:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:23:59.951+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:00.935 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:24:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:24:00.654+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29579 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:24:00.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:00.708+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:00.936 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:00.708+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:00.936 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:00.708+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:00.936 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:00.709+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:01.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:00.935+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:01.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:00.978+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:01.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:00.978+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:01.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:00.978+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:01.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:00.979+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:01.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:01.898+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:01.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:01.689+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:01.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:01.689+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:01.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:01.689+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:01.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:01.690+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:02.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:02.010+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:02.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:02.010+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:02.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:02.010+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:02.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:02.010+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:02.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:02.905+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:02.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:02.700+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:02.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:02.700+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:02.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:02.700+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:02.976 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:02.701+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:03.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:03.051+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:03.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:03.051+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:03.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:03.051+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:03.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:03.052+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:03.921 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:03.656+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:03.922 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:03.656+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:03.922 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:03.656+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:03.922 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:03.656+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:04.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:03.920+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:04.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:04.035+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:04.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:04.035+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:04.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:04.035+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:04.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:04.036+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:04.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:04.927+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:04.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:04.681+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:04.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:04.681+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:04.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:04.681+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:04.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:04.682+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:05.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:04.989+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:05.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:04.989+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:05.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:04.989+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:05.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:04.989+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:05.945 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:24:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:24:05.656+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29664 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:24:05.945 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:05.655+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:05.945 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:05.655+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:05.945 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:05.655+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:05.945 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:05.658+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:06.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:05.968+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:06.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:05.944+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:06.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:05.944+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:06.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:05.944+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:06.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:05.944+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:06.945 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:06.624+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:06.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:06.624+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:06.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:06.624+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:06.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:06.625+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:07.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:06.989+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:07.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:06.944+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:07.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:06.944+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:07.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:06.944+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:07.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:06.945+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:07.906 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:07.579+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:07.906 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:07.579+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:07.906 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:07.579+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:07.906 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:07.580+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:08.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:08.004+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:08.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:08.004+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:08.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:08.004+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:08.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:08.005+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:08.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:07.905+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:08.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:07.905+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:08.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:07.905+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:08.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:07.905+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:08.912 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:08.566+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:08.912 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:08.566+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:08.912 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:08.566+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:08.912 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:08.567+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:09.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:09.008+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:09.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:09.008+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:09.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:09.008+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:09.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:09.009+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:09.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:08.910+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:09.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:08.910+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:09.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:08.910+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:09.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:08.911+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:09.943 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:09.606+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:09.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:09.606+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:09.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:09.606+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:09.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:09.606+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:10.212 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:10.010+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:10.213 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:10.010+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:10.213 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:10.010+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:10.213 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:10.011+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:10.213 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:09.942+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:10.214 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:09.942+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:10.214 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:09.942+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:10.214 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:09.943+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:10.952 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:24:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:24:10.660+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29747 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:24:10.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:10.599+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:10.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:10.599+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:10.953 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:10.599+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:10.953 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:10.599+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:11.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:11.013+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:11.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:11.013+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:11.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:11.013+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:11.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:11.014+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:11.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:10.951+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:11.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:10.951+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:11.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:10.951+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:11.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:10.951+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:11.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:11.637+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:11.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:11.637+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:11.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:11.637+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:11.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:11.638+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:12.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:11.989+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:12.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:11.989+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:12.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:11.989+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:12.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:11.990+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:12.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:11.978+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:12.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:11.978+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:12.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:11.978+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:12.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:11.979+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:12.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:12.663+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:12.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:12.663+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:12.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:12.663+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:12.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:12.663+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:13.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:13.026+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:13.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:13.026+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:13.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:13.026+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:13.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:13.027+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:13.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:13.004+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:13.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:13.004+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:13.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:13.004+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:13.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:13.004+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:13.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:13.623+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:13.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:13.623+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:13.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:13.623+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:13.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:13.624+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:14.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:13.981+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:14.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:13.981+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:14.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:13.981+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:14.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:13.982+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:14.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:14.022+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:14.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:14.022+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:14.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:14.022+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:14.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:14.022+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:14.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:14.625+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:14.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:14.625+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:14.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:14.625+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:14.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:14.626+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:15.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:15.006+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:15.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:15.006+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:15.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:15.006+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:15.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:15.007+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:15.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:15.059+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:15.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:15.059+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:15.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:15.059+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:15.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:15.060+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:15.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:24:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:24:15.663+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29833 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:24:15.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:15.613+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:15.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:15.613+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:15.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:15.613+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:15.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:15.613+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:16.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:16.087+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:16.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:16.087+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:16.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:16.087+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:16.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:16.088+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:16.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:15.990+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:16.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:15.990+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:16.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:15.990+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:16.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:15.990+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:16.943 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:16.603+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:16.943 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:16.603+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:16.943 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:16.603+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:16.943 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:16.604+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:17.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:17.068+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:17.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:17.068+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:17.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:17.068+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:17.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:17.068+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:17.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:16.941+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:17.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:16.941+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:17.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:16.941+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:17.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:16.942+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:17.970 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:17.599+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:17.970 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:17.599+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:17.970 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:17.599+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:17.970 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:17.600+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:18.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:18.067+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:18.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:18.067+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:18.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:18.067+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:18.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:18.067+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:18.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:17.969+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:18.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:17.969+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:18.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:17.969+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:18.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:17.970+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:18.940 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:18.623+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:18.940 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:18.623+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:18.940 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:18.623+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:18.940 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:18.623+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:19.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:19.080+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:19.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:19.080+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:19.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:19.080+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:19.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:19.081+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:19.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:18.939+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:19.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:18.939+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:19.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:18.939+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:19.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:18.939+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:19.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:19.599+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:19.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:19.599+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:19.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:19.599+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:19.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:19.600+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:20.213 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:19.942+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:20.213 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:19.942+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:20.214 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:19.942+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:20.214 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:19.943+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:20.214 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:20.130+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:20.214 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:20.130+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:20.214 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:20.130+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:20.214 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:20.131+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:20.904 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:24:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:24:20.666+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29916 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:24:20.904 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:20.628+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:20.904 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:20.628+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:20.905 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:20.628+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:20.905 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:20.629+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:21.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:20.903+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:21.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:20.903+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:21.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:20.903+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:21.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:20.903+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:21.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:21.129+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:21.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:21.129+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:21.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:21.129+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:21.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:21.129+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:21.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:21.632+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:21.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:21.632+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:21.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:21.632+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:21.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:21.633+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:21.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:21.865+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:21.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:21.865+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:21.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:21.865+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:21.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:21.866+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:22.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:22.131+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:22.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:22.131+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:22.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:22.131+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:22.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:22.132+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:22.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:22.859+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:22.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:22.860+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:22.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:22.860+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:22.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:22.860+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:22.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:22.619+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:22.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:22.619+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:22.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:22.619+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:22.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:22.619+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:23.430 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:23.135+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:23.430 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:23.135+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:23.430 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:23.135+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:23.431 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:23.135+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:23.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:23.571+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:23.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:23.571+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:23.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:23.571+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:23.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:23.571+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:24.129 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:23.843+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:24.130 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:23.843+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:24.130 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:23.843+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:24.130 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:23.843+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:24.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:24.129+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:24.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:24.129+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:24.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:24.129+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:24.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:24.130+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:24.871 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:24.594+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:25.060 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:24.594+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:25.060 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:24.594+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:25.060 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:24.595+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:25.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:25.084+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:25.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:25.084+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:25.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:25.084+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:25.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:25.085+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:25.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:24.870+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:25.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:24.870+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:25.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:24.870+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:25.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:24.871+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:25.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:24:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:24:25.670+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30002 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:24:25.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:25.623+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:25.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:25.623+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:25.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:25.623+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:25.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:25.624+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:25.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:25.831+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:25.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:25.831+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:25.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:25.831+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:25.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:25.832+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:26.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:26.118+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:26.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:26.118+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:26.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:26.118+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:26.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:26.119+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:26.808 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:26.653+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:26.808 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:26.653+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:26.808 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:26.653+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:26.809 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:26.654+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:27.107 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:26.808+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:27.107 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:26.808+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:27.108 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:26.808+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:27.108 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:26.809+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:27.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:27.107+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:27.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:27.107+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:27.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:27.107+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:27.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:27.108+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:27.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:27.841+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:27.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:27.841+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:27.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:27.841+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:27.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:27.842+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:27.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:27.621+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:27.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:27.621+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:27.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:27.621+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:27.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:27.622+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:28.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:28.089+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:28.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:28.089+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:28.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:28.089+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:28.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:28.089+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:28.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:28.850+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:28.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:28.850+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:28.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:28.850+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:28.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:28.851+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:28.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:28.635+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:28.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:28.635+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:28.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:28.635+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:28.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:28.636+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:29.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:29.138+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:29.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:29.138+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:29.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:29.138+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:29.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:29.139+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:29.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:29.636+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:29.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:29.636+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:29.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:29.636+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:29.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:29.636+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:29.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:29.879+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:29.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:29.879+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:29.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:29.879+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:29.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:29.879+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:30.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:30.115+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:30.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:30.115+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:30.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:30.115+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:30.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:30.116+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:30.929 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:24:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:24:30.674+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30085 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:24:30.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:30.652+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:30.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:30.652+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:30.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:30.652+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:30.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:30.653+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:31.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:30.929+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:31.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:30.929+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:31.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:30.929+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:31.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:30.930+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:31.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:31.117+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:31.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:31.117+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:31.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:31.117+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:31.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:31.118+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:31.934 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:31.672+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:31.934 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:31.672+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:31.934 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:31.672+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:31.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:31.672+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:32.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:31.934+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:32.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:31.934+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:32.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:31.934+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:32.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:31.934+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:32.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:32.097+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:32.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:32.097+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:32.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:32.097+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:32.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:32.098+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:32.927 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:32.626+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:32.928 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:32.626+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:32.928 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:32.626+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:32.928 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:32.627+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:33.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:32.927+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:33.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:32.927+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:33.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:32.927+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:33.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:32.928+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:33.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:33.073+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:33.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:33.073+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:33.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:33.073+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:33.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:33.074+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:33.889 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:33.583+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:33.889 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:33.583+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:33.889 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:33.583+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:33.890 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:33.584+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:34.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:33.889+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:34.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:33.889+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:34.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:33.889+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:34.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:33.890+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:34.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:34.088+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:34.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:34.088+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:34.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:34.088+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:34.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:34.089+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:34.878 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:34.604+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:34.878 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:34.604+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:34.879 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:34.604+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:34.879 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:34.605+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:35.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:35.111+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:35.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:35.111+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:35.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:35.111+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:35.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:35.112+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:35.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:34.878+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:35.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:34.878+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:35.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:34.878+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:35.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:34.879+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:35.873 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:24:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:24:35.677+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30171 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:24:35.874 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:35.616+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:35.874 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:35.616+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:35.874 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:35.616+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:35.874 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:35.617+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:36.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:35.873+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:36.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:35.873+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:36.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:35.873+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:36.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:35.874+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:36.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:36.067+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:36.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:36.067+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:36.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:36.067+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:36.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:36.068+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:36.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:36.892+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:36.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:36.892+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:36.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:36.892+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:36.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:36.893+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:36.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:36.644+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:36.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:36.644+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:36.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:36.644+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:36.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:36.645+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:37.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:37.019+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:37.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:37.019+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:37.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:37.019+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:37.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:37.020+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:37.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:37.879+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:37.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:37.879+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:37.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:37.879+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:37.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:37.880+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:37.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:37.677+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:37.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:37.677+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:37.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:37.677+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:37.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:37.678+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:38.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:38.044+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:38.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:38.044+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:38.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:38.044+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:38.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:38.045+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:38.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:38.723+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:38.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:38.723+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:38.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:38.723+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:38.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:38.724+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:38.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:38.866+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:38.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:38.866+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:38.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:38.866+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:38.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:38.867+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:39.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:39.030+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:39.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:39.030+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:39.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:39.030+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:39.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:39.031+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:40.017 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:39.761+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:40.018 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:39.761+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:40.018 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:39.761+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:40.018 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:39.762+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:40.018 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:39.817+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:40.019 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:39.817+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:40.019 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:39.817+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:40.019 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:39.818+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:40.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:40.017+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:40.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:40.017+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:40.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:40.017+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:40.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:40.018+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:40.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:24:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:24:40.680+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30253 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:24:40.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:40.841+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:40.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:40.841+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:40.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:40.841+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:40.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:40.841+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:40.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:40.752+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:40.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:40.752+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:40.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:40.752+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:40.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:40.753+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:41.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:40.983+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:41.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:40.983+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:41.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:40.983+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:41.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:40.983+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:41.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:41.716+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:41.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:41.716+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:41.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:41.716+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:41.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:41.717+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:41.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:41.795+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:41.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:41.795+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:41.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:41.795+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:41.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:41.796+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:42.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:41.995+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:42.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:41.995+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:42.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:41.995+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:42.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:41.996+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:42.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:42.696+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:42.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:42.696+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:42.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:42.696+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:42.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:42.696+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:42.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:42.807+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:42.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:42.807+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:42.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:42.807+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:42.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:42.808+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:43.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:43.039+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:43.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:43.039+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:43.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:43.039+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:43.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:43.040+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:43.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:43.677+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:43.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:43.677+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:43.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:43.677+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:43.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:43.678+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:43.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:43.855+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:43.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:43.855+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:43.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:43.855+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:43.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:43.856+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:44.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:44.042+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:44.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:44.042+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:44.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:44.042+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:44.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:44.043+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:44.934 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:44.844+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:44.934 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:44.844+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:44.934 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:44.844+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:44.935 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:44.845+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:44.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:44.658+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:44.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:44.658+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:44.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:44.658+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:44.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:44.659+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:45.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:45.080+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:45.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:45.080+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:45.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:45.080+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:45.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:45.081+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:45.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:24:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:24:45.684+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30338 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:24:45.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:45.616+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:45.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:45.616+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:45.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:45.616+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:45.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:45.616+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:45.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:45.804+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:45.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:45.804+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:45.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:45.804+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:45.976 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:45.804+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:46.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:46.058+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:46.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:46.058+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:46.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:46.058+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:46.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:46.059+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:46.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:46.652+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:46.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:46.652+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:46.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:46.652+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:46.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:46.652+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:46.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:46.795+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:46.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:46.795+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:46.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:46.795+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:46.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:46.796+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:47.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:47.078+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:47.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:47.078+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:47.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:47.078+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:47.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:47.078+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:47.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:47.633+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:47.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:47.633+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:47.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:47.633+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:47.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:47.634+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:47.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:47.805+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:47.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:47.805+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:47.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:47.805+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:47.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:47.805+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:48.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:48.061+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:48.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:48.061+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:48.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:48.061+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:48.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:48.062+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:48.790 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:48.682+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:48.790 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:48.682+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:48.790 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:48.682+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:48.790 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:48.682+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:49.080 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:48.790+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:49.081 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:48.790+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:49.081 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:48.790+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:49.081 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:48.791+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:49.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:49.080+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:49.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:49.080+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:49.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:49.080+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:49.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:49.081+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:50.096 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:49.724+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:50.097 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:49.724+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:50.097 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:49.724+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:50.097 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:49.725+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:50.097 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:49.756+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:50.097 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:49.756+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:50.098 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:49.756+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:50.098 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:49.757+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:50.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:50.096+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:50.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:50.096+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:50.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:50.096+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:50.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:50.097+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:50.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:24:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:24:50.687+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30420 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:24:50.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:50.762+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:50.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:50.762+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:50.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:50.762+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:50.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:50.763+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:50.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:50.704+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:50.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:50.704+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:50.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:50.704+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:50.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:50.705+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:51.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:51.064+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:51.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:51.064+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:51.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:51.064+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:51.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:51.064+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:52.035 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:51.750+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:52.035 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:51.750+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:52.035 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:51.750+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:52.036 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:51.751+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:52.036 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:51.796+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:52.036 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:51.796+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:52.036 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:51.796+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:52.036 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:51.797+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:52.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:52.035+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:52.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:52.035+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:52.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:52.035+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:52.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:52.036+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:53.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:52.834+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:53.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:52.834+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:53.224 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:52.834+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:53.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:52.835+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:53.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:53.010+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:53.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:53.010+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:53.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:53.010+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:53.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:53.010+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:53.226 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:52.795+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:53.226 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:52.795+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:53.226 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:52.795+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:53.226 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:52.796+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:53.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:53.843+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:53.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:53.843+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:53.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:53.843+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:53.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:53.844+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:53.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:53.760+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:53.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:53.760+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:53.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:53.760+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:53.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:53.761+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:54.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:54.000+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:54.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:54.000+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:54.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:54.000+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:54.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:54.000+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:55.031 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:54.755+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:55.031 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:54.755+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:55.031 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:54.755+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:55.031 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:54.756+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:55.032 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:54.885+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:55.032 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:54.885+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:55.032 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:54.885+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:55.032 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:54.886+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:55.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:55.030+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:55.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:55.031+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:55.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:55.031+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:55.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:55.031+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:55.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:24:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:24:55.690+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30504 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:24:55.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:55.736+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:55.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:55.736+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:55.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:55.736+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:55.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:55.736+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:55.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:55.849+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:55.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:55.849+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:55.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:55.849+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:55.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:55.849+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:56.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:56.029+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:56.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:56.029+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:56.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:56.029+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:56.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:56.030+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:56.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:56.689+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:56.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:56.689+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:56.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:56.689+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:56.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:56.689+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:56.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:56.870+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:56.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:56.870+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:56.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:56.870+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:56.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:56.871+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:57.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:56.982+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:57.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:56.982+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:57.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:56.982+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:57.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:56.983+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:57.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:57.655+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:57.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:57.655+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:57.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:57.655+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:57.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:57.656+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:57.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:57.860+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:57.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:57.860+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:57.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:57.860+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:57.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:57.861+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:58.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:58.028+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:58.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:58.028+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:58.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:58.028+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:58.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:58.028+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:58.905 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:58.618+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:58.905 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:58.618+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:58.905 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:58.618+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:58.905 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:58.619+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:59.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:58.985+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:59.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:58.985+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:59.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:58.985+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:24:59.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:58.985+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:24:59.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:58.904+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:59.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:58.904+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:59.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:58.904+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:59.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:58.905+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:59.859+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:59.859+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:59.859+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:24:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:24:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:24:59.860+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:24:59.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:59.662+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:59.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:59.662+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:59.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:59.662+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:24:59.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:24:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:24:59.663+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:00.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:59.973+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:00.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:59.973+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:00.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:59.973+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:00.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:24:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:24:59.974+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:00.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:25:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:25:00.694+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30587 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:25:00.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:00.706+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:00.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:00.706+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:00.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:00.706+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:00.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:00.707+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:00.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:00.883+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:00.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:00.883+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:00.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:00.883+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:00.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:00.883+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:01.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:01.000+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:01.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:01.000+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:01.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:01.000+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:01.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:01.001+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:01.970 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:01.685+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:01.970 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:01.685+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:01.970 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:01.685+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:01.970 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:01.686+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:01.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:01.891+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:01.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:01.891+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:01.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:01.891+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:01.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:01.892+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:01.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:01.969+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:01.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:01.969+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:01.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:01.969+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:02.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:01.970+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:02.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:02.694+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:02.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:02.694+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:02.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:02.694+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:02.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:02.695+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:02.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:02.855+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:02.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:02.855+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:02.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:02.855+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:02.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:02.856+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:03.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:02.946+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:03.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:02.947+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:03.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:02.947+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:03.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:02.948+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:03.969 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:03.720+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:03.969 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:03.720+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:03.969 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:03.720+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:03.969 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:03.720+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:03.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:03.818+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:03.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:03.818+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:03.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:03.818+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:03.970 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:03.818+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:04.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:03.969+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:04.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:03.969+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:04.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:03.969+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:04.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:03.969+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:04.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:04.699+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:04.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:04.699+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:04.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:04.699+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:04.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:04.700+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:04.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:04.838+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:04.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:04.838+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:04.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:04.838+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:04.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:04.839+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:05.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:04.987+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:05.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:04.987+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:05.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:04.987+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:05.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:04.988+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:05.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:25:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:25:05.697+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30672 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:25:05.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:05.861+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:05.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:05.861+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:05.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:05.861+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:05.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:05.862+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:05.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:05.698+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:05.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:05.698+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:05.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:05.698+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:05.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:05.699+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:06.438 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:05.984+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:06.438 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:05.984+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:06.438 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:05.984+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:06.439 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:05.985+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:06.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:06.671+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:06.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:06.671+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:06.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:06.671+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:06.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:06.671+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:07.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:06.992+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:07.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:06.992+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:07.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:06.992+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:07.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:06.993+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:07.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:06.866+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:07.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:06.866+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:07.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:06.866+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:07.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:06.867+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:07.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:07.676+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:07.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:07.676+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:07.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:07.676+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:07.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:07.677+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:07.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:07.896+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:07.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:07.896+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:07.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:07.896+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:07.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:07.897+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:08.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:08.038+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:08.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:08.038+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:08.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:08.038+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:08.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:08.038+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:08.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:08.652+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:08.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:08.652+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:08.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:08.652+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:08.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:08.652+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:08.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:08.881+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:08.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:08.881+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:08.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:08.881+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:08.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:08.882+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:09.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:09.080+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:09.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:09.080+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:09.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:09.080+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:09.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:09.080+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:09.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:09.676+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:09.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:09.676+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:09.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:09.676+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:09.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:09.677+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:09.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:09.833+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:09.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:09.833+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:09.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:09.833+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:09.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:09.833+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:10.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:10.117+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:10.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:10.117+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:10.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:10.117+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:10.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:10.118+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:10.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:25:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:25:10.700+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30755 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:25:10.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:10.838+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:10.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:10.838+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:10.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:10.838+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:10.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:10.839+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:10.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:10.694+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:10.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:10.694+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:10.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:10.694+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:10.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:10.694+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:11.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:11.113+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:11.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:11.113+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:11.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:11.113+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:11.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:11.114+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:11.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:11.685+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:11.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:11.685+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:11.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:11.685+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:11.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:11.686+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:11.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:11.826+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:11.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:11.826+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:11.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:11.826+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:11.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:11.827+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:12.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:12.138+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:12.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:12.138+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:12.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:12.138+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:12.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:12.138+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:12.836 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:12.709+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:12.836 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:12.709+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:12.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:12.709+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:12.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:12.709+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:13.173 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:12.836+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:13.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:12.836+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:13.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:12.836+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:13.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:12.837+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:13.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:13.173+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:13.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:13.173+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:13.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:13.173+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:13.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:13.174+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:13.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:13.703+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:13.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:13.703+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:13.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:13.703+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:13.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:13.704+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:13.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:13.821+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:13.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:13.821+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:13.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:13.821+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:13.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:13.822+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:14.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:14.217+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:14.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:14.217+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:14.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:14.217+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:14.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:14.217+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:14.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:14.678+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:14.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:14.678+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:14.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:14.678+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:14.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:14.678+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:14.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:14.807+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:14.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:14.807+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:14.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:14.807+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:14.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:14.808+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:15.654 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:15.232+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:15.654 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:15.232+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:15.654 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:15.232+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:15.655 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:15.233+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:15.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:25:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:25:15.704+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30840 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:25:15.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:15.707+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:15.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:15.707+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:15.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:15.707+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:15.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:15.708+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:15.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:15.773+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:15.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:15.773+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:15.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:15.773+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:15.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:15.774+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:16.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:16.233+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:16.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:16.233+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:16.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:16.233+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:16.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:16.234+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:17.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:16.731+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:17.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:16.731+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:17.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:16.731+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:17.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:16.732+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:17.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:16.724+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:17.221 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:16.724+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:17.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:16.724+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:17.222 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:16.725+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:17.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:17.220+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:17.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:17.220+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:17.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:17.220+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:17.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:17.221+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:17.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:17.740+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:17.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:17.740+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:17.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:17.740+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:17.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:17.741+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:17.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:17.691+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:17.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:17.691+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:17.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:17.691+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:17.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:17.692+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:18.690 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:18.248+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:18.690 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:18.248+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:18.691 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:18.248+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:18.691 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:18.248+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:18.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:18.731+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:18.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:18.731+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:18.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:18.731+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:18.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:18.731+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:18.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:18.690+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:18.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:18.690+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:18.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:18.690+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:18.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:18.691+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:19.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:19.221+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:19.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:19.221+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:19.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:19.221+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:19.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:19.221+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:19.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:19.739+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:19.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:19.739+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:19.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:19.739+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:19.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:19.739+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:19.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:19.706+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:19.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:19.706+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:19.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:19.706+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:19.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:19.707+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:20.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:20.214+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:20.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:20.214+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:20.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:20.214+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:20.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:20.215+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:20.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:25:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:25:20.708+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30924 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:25:20.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:20.755+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:20.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:20.755+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:20.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:20.755+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:20.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:20.756+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:20.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:20.708+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:20.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:20.708+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:20.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:20.708+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:20.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:20.709+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:21.709 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:21.230+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:21.709 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:21.230+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:21.710 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:21.230+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:21.710 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:21.231+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:21.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:21.709+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:21.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:21.709+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:21.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:21.709+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:21.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:21.710+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:21.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:21.780+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:21.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:21.780+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:21.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:21.780+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:21.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:21.781+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:22.671 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:22.275+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:22.672 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:22.275+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:22.672 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:22.275+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:22.672 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:22.276+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:22.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:22.671+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:22.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:22.671+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:22.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:22.671+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:22.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:22.672+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:22.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:22.825+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:22.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:22.825+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:22.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:22.825+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:22.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:22.826+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:23.624 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:23.241+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:23.624 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:23.241+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:23.624 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:23.241+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:23.625 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:23.242+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:23.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:23.869+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:23.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:23.869+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:23.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:23.869+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:23.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:23.869+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:23.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:23.624+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:23.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:23.624+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:23.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:23.624+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:23.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:23.625+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:24.623 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:24.231+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:24.624 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:24.231+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:24.624 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:24.231+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:24.624 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:24.232+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:24.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:24.867+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:24.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:24.867+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:24.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:24.867+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:24.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:24.868+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:24.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:24.623+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:24.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:24.623+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:24.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:24.623+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:24.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:24.624+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:25.580 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:25.274+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:25.580 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:25.274+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:25.580 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:25.274+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:25.580 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:25.275+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:25.834 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:25:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:25:25.710+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 31009 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:25:25.834 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:25.578+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:25.835 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:25.578+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:25.835 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:25.578+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:25.835 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:25.579+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:26.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:25.906+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:26.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:25.907+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:26.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:25.907+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:26.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:25.908+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:26.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:26.290+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:26.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:26.290+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:26.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:26.290+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:26.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:26.290+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:26.878 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:26.545+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:26.878 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:26.545+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:26.878 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:26.545+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:26.878 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:26.547+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:27.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:26.876+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:27.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:26.876+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:27.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:26.876+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:27.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:26.878+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:27.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:27.507+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:27.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:27.507+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:27.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:27.507+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:27.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:27.507+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:27.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:27.318+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:27.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:27.318+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:27.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:27.318+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:27.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:27.319+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:28.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:27.909+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:28.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:27.909+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:28.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:27.909+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:28.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:27.910+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:28.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:28.363+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:28.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:28.363+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:28.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:28.363+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:28.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:28.364+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:28.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:28.525+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:28.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:28.525+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:28.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:28.525+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:28.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:28.526+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:29.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:28.909+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:29.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:28.909+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:29.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:28.909+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:29.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:28.910+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:29.538 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:29.388+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:29.538 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:29.388+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:29.539 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:29.388+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:29.539 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:29.389+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:29.885 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:29.537+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:29.885 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:29.537+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:29.885 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:29.537+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:29.885 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:29.538+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:30.217 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:29.884+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:30.218 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:29.884+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:30.218 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:29.884+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:30.218 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:29.885+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:30.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:30.429+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:30.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:30.429+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:30.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:30.429+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:30.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:30.429+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:30.836 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:25:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:25:30.714+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 31091 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:25:30.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:30.508+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:30.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:30.508+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:30.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:30.508+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:30.837 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:30.509+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:31.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:30.924+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:31.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:30.924+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:31.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:30.924+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:31.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:30.925+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:31.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:31.476+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:31.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:31.476+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:31.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:31.476+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:31.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:31.477+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:31.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:31.381+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:31.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:31.381+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:31.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:31.381+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:31.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:31.382+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:32.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:31.895+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:32.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:31.895+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:32.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:31.895+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:32.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:31.895+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:32.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:32.432+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:32.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:32.432+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:32.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:32.432+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:32.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:32.433+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:32.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:32.426+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:32.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:32.426+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:32.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:32.426+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:32.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:32.427+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:33.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:32.873+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:33.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:32.873+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:33.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:32.873+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:33.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:32.874+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:33.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:33.417+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:33.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:33.417+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:33.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:33.417+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:33.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:33.418+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:33.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:33.422+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:33.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:33.422+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:33.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:33.422+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:33.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:33.422+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:34.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:33.869+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:34.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:33.869+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:34.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:33.869+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:34.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:33.869+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:34.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:34.399+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:34.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:34.399+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:34.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:34.399+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:34.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:34.400+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:34.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:34.394+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:34.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:34.394+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:34.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:34.394+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:34.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:34.395+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:35.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:34.846+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:35.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:34.846+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:35.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:34.846+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:35.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:34.847+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:35.719 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:35.435+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:35.719 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:35.435+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:35.719 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:35.436+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:35.720 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:35.436+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:35.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:35.360+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:35.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:35.360+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:35.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:35.360+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:35.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:35.360+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:35.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:25:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:25:35.718+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 31169 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:25:35.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:35.848+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:35.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:35.848+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:35.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:35.848+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:35.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:35.849+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:36.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:36.411+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:36.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:36.411+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:36.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:36.411+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:36.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:36.412+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:36.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:36.317+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:36.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:36.317+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:36.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:36.317+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:36.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:36.318+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:37.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:36.855+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:37.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:36.855+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:37.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:36.855+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:37.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:36.857+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:37.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:37.410+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:37.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:37.410+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:37.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:37.410+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:37.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:37.411+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:37.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:37.316+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:37.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:37.316+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:37.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:37.316+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:37.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:37.317+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:38.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:37.893+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:38.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:37.893+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:38.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:37.893+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:38.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:37.894+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:38.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:38.363+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:38.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:38.363+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:38.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:38.363+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:38.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:38.363+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:38.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:38.394+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:38.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:38.394+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:38.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:38.394+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:38.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:38.395+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:39.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:38.844+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:39.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:38.844+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:39.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:38.844+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:39.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:38.845+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:39.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:39.433+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:39.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:39.433+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:39.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:39.433+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:39.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:39.433+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:39.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:39.380+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:39.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:39.380+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:39.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:39.380+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:39.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:39.380+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:40.218 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:39.818+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:40.218 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:39.818+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:40.218 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:39.818+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:40.218 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:39.818+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:40.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:40.417+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:40.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:40.417+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:40.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:40.417+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:40.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:40.418+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:40.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:40.371+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:40.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:40.371+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:40.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:40.371+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:40.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:40.371+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:40.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:25:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:25:40.721+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 31247 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:25:40.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:40.837+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:40.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:40.837+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:40.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:40.837+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:40.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:40.837+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:41.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:41.445+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:41.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:41.445+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:41.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:41.445+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:41.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:41.446+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:41.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:41.386+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:41.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:41.386+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:41.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:41.386+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:41.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:41.387+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:42.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:41.792+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:42.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:41.792+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:42.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:41.792+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:42.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:41.792+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:42.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:42.472+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:42.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:42.472+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:42.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:42.472+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:42.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:42.473+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:42.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:42.422+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:42.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:42.422+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:42.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:42.422+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:42.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:42.422+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:43.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:42.820+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:43.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:42.820+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:43.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:42.820+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:43.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:42.820+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:43.629 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:43.501+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:43.630 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:43.501+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:43.630 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:43.501+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:43.630 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:43.502+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:43.630 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:43.453+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:43.630 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:43.453+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:43.630 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:43.453+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:43.631 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:43.454+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:43.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:43.859+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:43.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:43.859+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:43.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:43.859+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:43.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:43.859+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:44.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:44.505+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:44.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:44.505+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:44.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:44.505+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:44.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:44.505+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:44.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:44.457+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:44.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:44.457+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:44.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:44.457+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:44.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:44.458+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:45.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:44.905+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:45.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:44.905+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:45.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:44.905+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:45.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:44.907+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:45.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:45.462+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:45.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:45.462+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:45.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:45.462+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:45.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:45.463+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:45.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:45.414+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:45.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:45.414+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:45.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:45.414+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:45.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:45.415+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:46.224 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:25:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:25:45.726+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 31326 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:25:46.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:45.865+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:46.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:45.865+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:46.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:45.865+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:46.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:45.866+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:46.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:46.464+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:46.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:46.464+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:46.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:46.464+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:46.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:46.465+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:46.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:46.425+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:46.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:46.425+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:46.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:46.425+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:46.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:46.426+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:47.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:46.852+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:47.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:46.852+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:47.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:46.852+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:47.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:46.853+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:47.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:47.404+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:47.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:47.404+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:47.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:47.404+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:47.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:47.405+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:47.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:47.465+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:47.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:47.465+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:47.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:47.465+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:47.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:47.466+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:48.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:47.815+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:48.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:47.815+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:48.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:47.815+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:48.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:47.816+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:48.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:48.450+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:48.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:48.450+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:48.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:48.450+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:48.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:48.451+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:48.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:48.422+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:48.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:48.422+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:48.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:48.422+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:48.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:48.423+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:49.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:48.815+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:49.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:48.815+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:49.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:48.815+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:49.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:48.816+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:49.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:49.414+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:49.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:49.414+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:49.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:49.414+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:49.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:49.414+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:49.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:49.450+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:49.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:49.450+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:49.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:49.450+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:49.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:49.451+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:50.219 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:49.835+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:50.219 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:49.835+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:50.219 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:49.835+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:50.219 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:49.836+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:50.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:50.452+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:50.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:50.452+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:50.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:50.452+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:50.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:50.453+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:50.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:50.441+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:50.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:50.441+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:50.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:50.441+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:50.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:50.442+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:51.224 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:25:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:25:50.730+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 31401 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:25:51.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:50.790+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:51.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:50.790+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:51.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:50.790+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:51.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:50.791+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:51.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:51.467+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:51.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:51.467+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:51.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:51.467+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:51.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:51.467+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:51.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:51.486+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:51.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:51.486+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:51.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:51.486+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:51.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:51.487+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:52.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:51.811+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:52.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:51.811+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:52.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:51.811+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:52.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:51.811+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:52.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:52.419+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:52.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:52.419+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:52.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:52.419+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:52.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:52.421+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:52.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:52.521+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:52.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:52.521+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:52.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:52.521+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:52.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:52.522+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:53.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:52.811+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:53.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:52.811+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:53.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:52.811+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:53.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:52.813+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:53.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:53.489+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:53.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:53.489+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:53.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:53.489+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:53.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:53.490+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:53.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:53.411+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:53.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:53.411+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:53.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:53.411+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:53.726 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:53.413+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:54.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:53.852+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:54.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:53.852+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:54.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:53.852+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:54.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:53.853+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:54.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:54.456+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:54.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:54.456+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:54.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:54.456+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:54.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:54.456+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:54.446+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:54.446+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:54.446+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:54.447+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:55.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:54.840+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:55.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:54.840+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:55.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:54.840+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:55.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:54.841+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:55.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:55.462+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:55.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:55.462+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:55.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:55.462+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:55.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:55.463+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:55.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:55.405+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:55.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:55.405+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:55.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:55.405+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:55.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:55.406+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:56.224 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:25:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:25:55.733+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 31480 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:25:56.227 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:55.823+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:56.227 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:55.823+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:56.227 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:55.823+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:56.228 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:55.824+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:56.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:56.478+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:56.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:56.478+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:56.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:56.478+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:56.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:56.479+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:56.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:56.363+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:56.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:56.363+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:56.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:56.363+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:56.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:56.363+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:57.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:56.853+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:57.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:56.853+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:57.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:56.853+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:57.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:56.854+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:57.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:57.492+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:57.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:57.492+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:57.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:57.492+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:57.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:57.492+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:57.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:57.395+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:57.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:57.395+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:57.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:57.395+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:57.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:57.396+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:58.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:57.880+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:58.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:57.880+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:58.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:57.880+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:58.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:57.881+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:58.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:58.535+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:58.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:58.535+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:58.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:58.535+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:58.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:58.536+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:25:58.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:58.370+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:58.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:58.370+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:58.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:58.370+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:58.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:58.371+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:59.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:58.916+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:59.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:58.916+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:59.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:58.916+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:25:59.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:58.917+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:59.561 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:59.410+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:59.562 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:59.410+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:59.562 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:59.410+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:25:59.562 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:25:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:25:59.411+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:25:59.933 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:59.561+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:59.933 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:59.561+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:59.933 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:59.561+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:25:59.933 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:25:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:25:59.562+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:00.219 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:59.932+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:00.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:59.932+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:00.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:59.933+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:00.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:25:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:25:59.933+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:00.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:00.410+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:00.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:00.410+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:00.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:00.410+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:00.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:00.411+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:00.859 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:26:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:26:00.737+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 31561 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:26:00.859 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:00.556+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:00.859 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:00.556+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:00.859 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:00.556+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:00.860 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:00.556+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:01.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:00.959+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:01.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:00.959+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:01.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:00.959+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:01.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:00.960+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:01.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:01.535+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:01.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:01.535+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:01.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:01.535+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:01.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:01.536+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:01.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:01.384+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:01.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:01.384+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:01.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:01.384+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:01.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:01.384+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:02.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:01.967+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:02.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:01.967+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:02.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:01.967+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:02.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:01.968+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:02.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:02.528+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:02.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:02.528+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:02.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:02.528+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:02.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:02.529+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:02.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:02.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:02.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:02.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:02.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:02.368+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:02.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:02.369+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:03.363 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:02.991+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:03.363 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:02.991+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:03.363 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:02.991+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:03.363 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:02.992+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:03.630 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:03.499+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:03.631 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:03.499+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:03.631 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:03.499+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:03.631 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:03.500+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:03.631 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:03.362+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:03.631 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:03.363+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:03.631 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:03.363+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:03.632 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:03.363+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:04.412 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:04.001+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:04.413 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:04.001+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:04.413 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:04.001+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:04.413 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:04.002+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:04.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:04.492+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:04.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:04.492+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:04.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:04.492+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:04.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:04.494+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:04.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:04.412+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:04.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:04.412+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:04.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:04.412+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:04.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:04.413+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:05.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:05.026+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:05.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:05.026+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:05.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:05.026+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:05.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:05.028+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:05.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:05.472+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:05.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:05.472+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:05.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:05.472+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:05.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:05.472+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:05.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:05.394+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:05.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:05.394+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:05.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:05.394+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:05.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:05.394+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:06.224 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:26:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:26:05.741+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 31643 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:26:06.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:05.983+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:06.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:05.983+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:06.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:05.983+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:06.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:05.983+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:06.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:06.430+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:06.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:06.430+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:06.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:06.430+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:06.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:06.431+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:06.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:06.350+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:06.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:06.350+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:06.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:06.350+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:06.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:06.350+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:07.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:07.012+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:07.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:07.012+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:07.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:07.012+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:07.338 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:07.012+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:07.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:07.337+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:07.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:07.337+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:07.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:07.337+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:07.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:07.338+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:07.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:07.416+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:07.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:07.416+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:07.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:07.416+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:07.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:07.417+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:08.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:07.965+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:08.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:07.965+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:08.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:07.965+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:08.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:07.965+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:08.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:08.312+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:08.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:08.312+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:08.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:08.312+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:08.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:08.312+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:08.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:08.395+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:08.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:08.395+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:08.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:08.395+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:08.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:08.396+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:09.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:08.948+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:09.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:08.948+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:09.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:08.948+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:09.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:08.949+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:09.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:09.352+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:09.727 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:09.352+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:09.727 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:09.352+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:09.727 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:09.353+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:09.728 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:09.339+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:09.728 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:09.339+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:09.728 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:09.339+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:09.728 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:09.340+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:10.220 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:09.930+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:10.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:09.930+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:10.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:09.930+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:10.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:09.931+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:10.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:10.323+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:10.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:10.323+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:10.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:10.323+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:10.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:10.324+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:10.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:10.386+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:10.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:10.386+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:10.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:10.386+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:10.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:10.387+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:11.224 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:26:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:26:10.745+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 31727 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:26:11.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:10.898+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:11.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:10.898+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:11.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:10.898+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:11.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:10.899+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:11.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:11.280+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:11.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:11.280+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:11.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:11.280+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:11.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:11.280+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:11.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:11.406+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:11.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:11.406+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:11.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:11.406+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:11.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:11.407+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:12.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:11.921+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:12.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:11.921+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:12.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:11.921+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:12.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:11.922+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:12.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:12.324+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:12.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:12.324+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:12.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:12.324+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:12.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:12.324+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:12.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:12.436+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:12.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:12.436+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:12.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:12.436+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:12.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:12.436+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:13.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:12.964+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:13.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:12.964+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:13.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:12.964+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:13.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:12.964+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:13.631 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:13.438+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:13.631 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:13.438+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:13.631 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:13.438+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:13.631 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:13.439+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:13.632 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:13.340+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:13.632 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:13.340+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:13.632 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:13.340+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:13.632 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:13.340+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:14.227 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:13.949+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:14.227 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:13.949+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:14.227 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:13.949+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:14.228 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:13.950+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:14.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:14.368+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:14.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:14.368+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:14.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:14.368+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:14.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:14.369+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:14.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:14.396+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:14.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:14.396+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:14.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:14.396+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:14.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:14.396+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:15.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:14.914+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:15.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:14.914+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:15.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:14.914+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:15.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:14.915+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:15.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:15.366+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:15.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:15.366+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:15.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:15.366+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:15.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:15.366+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:15.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:15.347+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:15.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:15.347+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:15.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:15.347+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:15.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:15.348+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:16.224 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:26:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:26:15.748+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 31812 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:26:16.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:15.899+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:16.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:15.899+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:16.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:15.899+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:16.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:15.900+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:16.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:16.356+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:16.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:16.356+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:16.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:16.356+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:16.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:16.357+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:16.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:16.332+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:16.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:16.332+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:16.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:16.332+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:16.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:16.333+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:17.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:16.898+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:17.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:16.898+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:17.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:16.898+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:17.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:16.898+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:17.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:17.380+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:17.819 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:17.380+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:17.819 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:17.380+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:17.819 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:17.381+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:17.820 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:17.327+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:17.820 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:17.327+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:17.820 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:17.327+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:17.820 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:17.328+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:18.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:17.873+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:18.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:17.873+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:18.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:17.873+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:18.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:17.874+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:18.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:18.415+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:18.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:18.415+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:18.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:18.415+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:18.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:18.416+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:18.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:18.342+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:18.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:18.342+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:18.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:18.342+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:18.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:18.343+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:19.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:18.923+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:19.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:18.923+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:19.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:18.923+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:19.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:18.924+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:19.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:19.389+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:19.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:19.389+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:19.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:19.389+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:19.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:19.390+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:19.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:19.320+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:19.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:19.320+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:19.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:19.320+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:19.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:19.321+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:20.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:19.942+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:20.221 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:19.942+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:20.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:19.942+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:20.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:19.943+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:20.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:20.304+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:20.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:20.304+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:20.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:20.304+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:20.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:20.305+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:20.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:20.347+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:20.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:20.347+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:20.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:20.347+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:20.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:20.348+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:21.224 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:26:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:26:20.752+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 31896 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:26:21.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:20.936+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:21.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:20.936+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:21.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:20.936+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:21.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:20.936+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:21.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:21.272+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:21.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:21.272+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:21.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:21.272+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:21.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:21.273+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:21.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:21.337+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:21.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:21.337+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:21.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:21.337+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:21.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:21.337+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:22.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:21.907+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:22.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:21.907+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:22.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:21.907+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:22.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:21.908+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:22.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:22.320+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:22.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:22.320+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:22.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:22.320+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:22.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:22.321+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:22.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:22.303+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:22.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:22.303+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:22.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:22.303+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:22.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:22.303+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:23.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:22.912+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:23.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:22.912+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:23.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:22.912+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:23.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:22.913+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:23.631 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:23.311+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:23.632 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:23.311+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:23.632 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:23.311+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:23.632 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:23.311+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:23.632 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:23.323+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:23.633 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:23.323+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:23.633 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:23.323+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:23.633 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:23.324+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:24.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:23.923+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:24.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:23.923+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:24.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:23.923+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:24.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:23.924+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:24.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:24.314+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:24.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:24.314+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:24.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:24.314+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:24.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:24.315+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:24.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:24.276+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:24.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:24.276+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:24.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:24.276+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:24.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:24.276+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:25.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:24.970+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:25.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:24.970+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:25.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:24.970+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:25.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:24.971+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:25.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:25.289+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:25.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:25.289+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:25.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:25.289+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:25.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:25.290+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:25.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:25.285+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:25.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:25.285+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:25.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:25.285+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:25.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:25.286+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:26.006 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:26:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:26:25.756+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 31981 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:26:26.007 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:26.005+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:26.007 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:26.005+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:26.007 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:26.005+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:26.007 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:26.006+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:26.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:26.259+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:26.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:26.259+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:26.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:26.259+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:26.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:26.260+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:26.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:26.289+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:26.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:26.289+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:26.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:26.289+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:26.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:26.290+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:27.279 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:27.213+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:27.280 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:27.213+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:27.280 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:27.213+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:27.280 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:27.214+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:27.280 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:27.012+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:27.280 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:27.012+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:27.281 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:27.012+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:27.281 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:27.013+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:27.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:27.279+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:27.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:27.279+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:27.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:27.279+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:27.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:27.280+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:28.256 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:28.165+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:28.256 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:28.165+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:28.257 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:28.165+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:28.257 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:28.166+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:28.257 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:27.998+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:28.257 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:27.998+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:28.257 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:27.998+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:28.257 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:27.999+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:28.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:28.256+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:28.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:28.256+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:28.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:28.256+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:28.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:28.257+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:29.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:29.040+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:29.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:29.040+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:29.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:29.040+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:29.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:29.041+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:29.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:29.165+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:29.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:29.165+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:29.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:29.165+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:29.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:29.166+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:29.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:29.256+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:29.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:29.256+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:29.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:29.256+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:29.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:29.257+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:30.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:30.202+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:30.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:30.202+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:30.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:30.202+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:30.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:30.203+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:30.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:30.211+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:30.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:30.211+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:30.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:30.211+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:30.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:30.211+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:30.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:30.019+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:30.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:30.019+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:30.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:30.019+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:30.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:30.020+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:31.068 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:26:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:26:30.760+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 32065 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:26:31.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:31.188+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:31.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:31.188+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:31.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:31.188+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:31.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:31.189+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:31.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:31.178+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:31.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:31.178+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:31.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:31.178+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:31.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:31.179+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:31.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:31.067+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:31.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:31.067+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:31.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:31.067+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:31.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:31.068+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:32.454 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:32.210+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:32.454 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:32.210+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:32.454 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:32.210+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:32.455 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:32.211+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:32.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:32.030+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:32.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:32.030+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:32.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:32.030+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:32.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:32.031+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:32.456 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:32.197+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:32.456 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:32.197+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:32.456 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:32.197+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:32.456 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:32.198+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:33.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:33.204+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:33.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:33.204+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:33.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:33.204+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:33.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:33.205+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:33.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:33.058+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:33.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:33.058+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:33.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:33.058+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:33.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:33.059+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:33.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:33.231+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:33.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:33.231+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:33.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:33.231+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:33.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:33.232+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:34.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:34.189+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:34.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:34.189+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:34.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:34.189+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:34.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:34.190+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:34.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:34.208+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:34.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:34.208+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:34.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:34.208+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:34.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:34.209+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:34.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:34.050+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:34.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:34.050+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:34.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:34.050+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:34.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:34.051+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:35.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:35.088+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:35.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:35.088+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:35.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:35.088+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:35.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:35.088+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:35.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:35.177+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:35.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:35.177+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:35.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:35.177+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:35.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:35.178+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:35.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:35.217+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:35.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:35.217+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:35.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:35.217+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:35.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:35.218+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:35.784 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:26:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:26:35.763+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 32150 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:26:36.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:36.060+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:36.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:36.060+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:36.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:36.060+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:36.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:36.061+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:36.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:36.259+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:36.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:36.259+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:36.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:36.259+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:36.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:36.260+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:36.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:36.164+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:36.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:36.164+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:36.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:36.164+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:36.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:36.165+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:37.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:37.260+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:37.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:37.260+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:37.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:37.260+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:37.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:37.261+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:37.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:37.123+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:37.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:37.123+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:37.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:37.123+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:37.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:37.124+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:37.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:37.061+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:37.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:37.061+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:37.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:37.061+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:37.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:37.062+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:38.299 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:38.024+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:38.299 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:38.024+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:38.299 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:38.024+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:38.300 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:38.025+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:38.300 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:38.161+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:38.300 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:38.161+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:38.300 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:38.161+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:38.300 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:38.162+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:38.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:38.299+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:38.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:38.299+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:38.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:38.299+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:38.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:38.299+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:39.297 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:39.028+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:39.298 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:39.028+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:39.298 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:39.028+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:39.298 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:39.028+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:39.298 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:39.158+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:39.298 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:39.158+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:39.298 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:39.158+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:39.299 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:39.159+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:39.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:39.297+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:39.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:39.297+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:39.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:39.297+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:39.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:39.298+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:40.303 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:40.004+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:40.303 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:40.004+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:40.303 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:40.004+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:40.303 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:40.005+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:40.303 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:40.189+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:40.304 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:40.189+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:40.304 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:40.189+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:40.304 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:40.190+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:40.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:40.302+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:40.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:40.302+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:40.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:40.302+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:40.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:40.303+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:41.221 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:26:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:26:40.767+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 32234 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:26:41.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:40.955+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:41.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:40.955+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:41.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:40.955+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:41.222 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:40.956+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:41.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:41.221+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:41.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:41.221+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:41.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:41.221+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:41.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:41.222+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:41.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:41.313+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:41.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:41.313+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:41.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:41.313+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:41.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:41.314+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:42.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:41.933+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:42.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:41.933+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:42.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:41.933+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:42.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:41.933+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:42.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:42.315+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:42.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:42.315+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:42.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:42.315+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:42.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:42.316+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:42.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:42.238+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:42.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:42.238+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:42.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:42.238+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:42.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:42.239+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:43.277 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:42.982+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:43.278 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:42.982+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:43.278 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:42.982+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:43.278 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:42.983+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:43.633 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:43.353+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:43.633 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:43.353+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:43.633 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:43.353+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:43.633 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:43.354+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:43.634 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:43.277+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:43.634 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:43.277+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:43.634 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:43.277+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:43.634 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:43.278+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:44.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:43.961+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:44.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:43.961+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:44.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:43.961+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:44.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:43.962+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:44.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:44.308+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:44.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:44.308+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:44.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:44.308+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:44.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:44.309+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:44.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:44.375+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:44.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:44.375+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:44.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:44.375+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:44.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:44.376+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:45.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:44.943+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:45.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:44.943+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:45.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:44.943+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:45.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:44.944+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:45.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:45.409+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:45.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:45.409+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:45.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:45.409+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:45.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:45.409+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:45.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:45.297+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:45.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:45.297+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:45.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:45.297+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:45.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:45.297+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:46.224 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:26:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:26:45.771+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 32319 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:26:46.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:45.974+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:46.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:45.974+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:46.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:45.974+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:46.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:45.975+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:46.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:46.316+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:46.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:46.316+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:46.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:46.316+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:46.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:46.317+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:46.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:46.426+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:46.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:46.426+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:46.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:46.426+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:46.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:46.427+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:47.323 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:47.018+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:47.323 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:47.018+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:47.323 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:47.018+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:47.324 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:47.018+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:47.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:47.465+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:47.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:47.465+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:47.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:47.465+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:47.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:47.466+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:47.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:47.322+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:47.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:47.322+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:47.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:47.322+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:47.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:47.322+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:48.368 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:48.055+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:48.368 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:48.055+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:48.369 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:48.055+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:48.369 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:48.056+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:48.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:48.464+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:48.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:48.464+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:48.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:48.464+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:48.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:48.465+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:48.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:48.367+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:48.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:48.367+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:48.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:48.367+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:48.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:48.368+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:49.362 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:49.047+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:49.362 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:49.047+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:49.362 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:49.047+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:49.362 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:49.048+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:49.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:49.454+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:49.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:49.454+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:49.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:49.454+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:49.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:49.455+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:49.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:49.361+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:49.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:49.361+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:49.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:49.361+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:49.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:49.362+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:50.387 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:50.055+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:50.388 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:50.055+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:50.388 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:50.055+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:50.388 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:50.056+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:50.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:50.386+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:50.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:50.386+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:50.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:50.386+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:50.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:50.387+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:50.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:50.419+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:50.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:50.419+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:50.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:50.419+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:50.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:50.419+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:51.034 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:26:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:26:50.774+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 32403 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:26:51.363 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:51.033+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:51.364 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:51.033+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:51.364 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:51.033+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:51.364 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:51.034+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:51.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:51.362+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:51.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:51.362+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:51.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:51.362+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:51.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:51.363+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:51.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:51.404+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:51.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:51.404+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:51.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:51.404+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:51.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:51.405+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:52.359 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:52.079+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:52.360 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:52.079+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:52.360 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:52.079+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:52.360 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:52.080+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:52.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:52.393+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:52.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:52.393+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:52.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:52.393+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:52.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:52.394+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:52.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:52.358+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:52.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:52.358+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:52.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:52.358+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:52.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:52.359+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:53.372 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:53.058+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:53.372 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:53.058+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:53.372 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:53.058+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:53.372 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:53.060+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:53.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:53.405+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:53.634 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:53.405+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:53.634 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:53.405+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:53.634 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:53.406+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:53.634 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:53.370+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:53.634 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:53.370+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:53.634 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:53.370+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:53.634 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:53.371+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:54.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:54.051+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:54.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:54.051+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:54.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:54.051+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:54.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:54.051+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:54.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:54.369+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:54.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:54.369+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:54.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:54.369+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:54.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:54.370+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:54.379+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:54.379+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:54.379+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:54.380+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:55.403 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:55.069+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:55.403 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:55.069+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:55.403 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:55.069+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:55.403 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:55.070+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:55.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:55.401+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:55.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:55.402+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:55.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:55.402+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:55.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:55.403+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:55.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:55.425+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:55.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:55.425+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:55.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:55.425+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:55.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:55.425+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:56.094 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:26:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:26:55.778+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 32488 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:26:56.427 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:56.093+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:56.428 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:56.093+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:56.428 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:56.093+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:56.428 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:56.094+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:56.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:56.426+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:56.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:56.426+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:56.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:56.426+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:56.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:56.427+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:56.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:56.443+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:56.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:56.443+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:56.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:56.443+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:56.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:56.444+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:56.892 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-01-23T09:26:56.898 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-23T09:26:56.898 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-01-23T09:26:56.899 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-01-23T09:26:56.973 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-01-23T09:26:56.984 INFO:tasks.cephadm:Teardown begin 2024-01-23T09:26:56.985 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/cephadm.py", line 1845, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-01-23T09:26:56.990 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-23T09:26:57.031 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:26:57.032 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:26:57.424 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:57.102+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:57.424 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:57.102+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:57.425 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:57.102+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:57.425 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:57.102+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:57.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:57.423+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:57.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:57.423+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:57.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:57.423+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:57.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:57.424+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:57.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:57.445+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:57.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:57.445+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:57.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:57.445+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:57.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:57.445+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:58.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:58.077+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:58.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:58.077+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:58.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:58.077+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:58.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:58.077+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:58.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:58.473+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:58.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:58.473+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:58.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:58.473+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:58.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:58.474+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:58.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:58.491+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:58.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:58.491+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:58.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:58.491+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:58.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:58.492+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:59.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:59.125+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:59.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:59.125+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:59.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:59.125+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:26:59.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:26:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:26:59.126+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:26:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:59.510+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:59.510+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:59.510+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:26:59.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:26:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:26:59.511+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:26:59.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:59.475+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:59.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:59.475+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:59.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:59.475+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:26:59.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:26:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:26:59.476+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:00.092 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:27:00.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:00.158+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:00.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:00.158+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:00.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:00.158+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:00.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:00.159+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:00.783 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:00.504+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:00.783 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:00.504+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:00.783 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:00.504+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:00.783 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:00.505+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:00.784 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:00.489+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:00.784 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:00.489+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:00.784 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:00.489+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:00.784 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:00.489+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:01.183 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:27:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:27:00.781+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 32572 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:27:01.468 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:01.182+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:01.468 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:01.182+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:01.469 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:01.182+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:01.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:01.183+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:01.727 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:01.468+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:01.728 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:01.468+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:01.728 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:01.468+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:01.728 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:01.468+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:01.728 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:01.467+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:01.728 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:01.467+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:01.728 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:01.467+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:01.729 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:01.468+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:02.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:02.193+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:02.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:02.193+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:02.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:02.193+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:02.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:02.194+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:02.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:02.425+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:02.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:02.425+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:02.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:02.425+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:02.477 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:02.426+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:02.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:02.489+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:02.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:02.489+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:02.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:02.489+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:02.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:02.490+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:03.422 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:03.149+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:03.422 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:03.149+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:03.423 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:03.149+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:03.423 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:03.150+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:03.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:03.421+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:03.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:03.421+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:03.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:03.421+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:03.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:03.422+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:03.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:03.493+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:03.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:03.494+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:03.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:03.494+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:03.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:03.494+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:04.397 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:04.118+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:04.397 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:04.118+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:04.397 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:04.118+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:04.397 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:04.118+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:04.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:04.396+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:04.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:04.396+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:04.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:04.396+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:04.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:04.396+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:04.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:04.490+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:04.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:04.490+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:04.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:04.490+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:04.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:04.490+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:05.401 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:05.129+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:05.401 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:05.129+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:05.401 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:05.129+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:05.401 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:05.129+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:05.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:05.522+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:05.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:05.522+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:05.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:05.522+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:05.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:05.522+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:05.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:05.399+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:05.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:05.399+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:05.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:05.400+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:05.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:05.401+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:06.116 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:27:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:27:05.785+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 32657 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:27:06.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:06.115+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:06.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:06.115+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:06.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:06.115+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:06.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:06.116+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:06.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:06.364+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:06.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:06.364+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:06.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:06.364+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:06.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:06.365+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:06.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:06.523+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:06.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:06.523+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:06.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:06.523+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:06.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:06.523+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:07.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:07.146+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:07.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:07.146+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:07.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:07.146+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:07.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:07.147+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:07.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:07.345+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:07.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:07.345+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:07.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:07.345+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:07.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:07.346+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:07.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:07.560+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:07.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:07.560+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:07.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:07.560+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:07.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:07.561+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:08.388 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:08.098+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:08.388 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:08.098+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:08.388 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:08.098+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:08.388 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:08.098+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:08.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:08.388+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:08.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:08.388+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:08.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:08.388+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:08.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:08.388+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:08.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:08.565+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:08.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:08.565+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:08.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:08.565+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:08.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:08.565+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:09.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:27:09.093 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:27:09.351 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:09.090+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:09.351 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:09.090+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:09.351 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:09.090+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:09.352 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:09.090+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:09.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:09.351+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:09.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:09.351+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:09.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:09.351+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:09.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:09.352+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:09.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:09.538+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:09.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:09.538+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:09.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:09.538+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:09.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:09.538+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:10.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:10.083+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:10.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:10.083+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:10.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:10.083+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:10.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:10.084+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:10.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:10.302+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:10.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:10.302+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:10.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:10.302+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:10.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:10.303+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:10.791 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:10.511+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:10.791 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:10.511+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:10.791 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:10.511+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:10.791 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:10.512+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:11.047 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:27:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:27:10.790+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 32741 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:27:11.334 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:11.047+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:11.335 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:11.047+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:11.335 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:11.047+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:11.335 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:11.048+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:11.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:11.334+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:11.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:11.334+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:11.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:11.334+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:11.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:11.335+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:11.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:11.536+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:11.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:11.536+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:11.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:11.536+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:11.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:11.537+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:12.159 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:27:12.390 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:12.286+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:12.390 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:12.286+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:12.391 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:12.286+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:12.391 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:12.287+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:12.391 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:12.058+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:12.391 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:12.058+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:12.391 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:12.058+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:12.391 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:12.059+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:12.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:12.506+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:12.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:12.506+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:12.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:12.506+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:12.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:12.506+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:13.319 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:13.039+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:13.319 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:13.039+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:13.319 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:13.039+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:13.319 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:13.040+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:13.635 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:13.318+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:13.635 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:13.318+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:13.635 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:13.318+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:13.636 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:13.319+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:13.636 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:13.493+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:13.636 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:13.493+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:13.636 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:13.493+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:13.636 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:13.494+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:14.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:14.285+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:14.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:14.285+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:14.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:14.285+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:14.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:14.285+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:14.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:14.059+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:14.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:14.059+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:14.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:14.059+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:14.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:14.059+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:14.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:14.485+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:14.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:14.485+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:14.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:14.485+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:14.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:14.486+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:15.294 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:15.032+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:15.294 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:15.032+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:15.294 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:15.032+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:15.294 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:15.033+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:15.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:15.293+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:15.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:15.293+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:15.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:15.293+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:15.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:15.294+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:15.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:15.482+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:15.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:15.482+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:15.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:15.482+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:15.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:15.483+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:16.224 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:27:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:27:15.794+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 32826 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:27:16.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:16.024+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:16.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:16.024+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:16.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:16.024+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:16.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:16.024+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:16.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:16.279+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:16.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:16.279+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:16.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:16.279+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:16.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:16.280+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:16.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:16.524+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:16.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:16.524+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:16.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:16.524+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:16.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:16.525+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:17.323 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:17.050+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:17.323 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:17.050+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:17.323 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:17.050+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:17.323 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:17.051+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:17.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:17.531+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:17.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:17.531+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:17.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:17.531+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:17.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:17.531+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:17.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:17.322+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:17.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:17.322+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:17.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:17.322+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:17.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:17.323+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:18.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:18.301+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:18.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:18.301+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:18.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:18.301+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:18.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:18.301+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:18.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:18.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:18.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:18.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:18.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:18.053+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:18.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:18.054+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:18.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:18.525+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:18.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:18.525+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:18.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:18.525+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:18.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:18.526+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:19.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:19.062+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:19.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:19.062+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:19.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:19.062+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:19.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:19.063+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:19.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:19.293+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:19.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:19.293+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:19.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:19.293+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:19.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:19.295+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:19.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:19.482+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:19.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:19.482+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:19.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:19.482+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:19.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:19.482+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:20.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:20.285+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:20.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:20.285+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:20.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:20.285+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:20.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:20.286+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:20.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:20.095+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:20.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:20.095+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:20.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:20.095+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:20.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:20.096+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:20.799 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:20.509+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:20.799 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:20.509+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:20.799 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:20.509+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:20.799 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:20.509+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:21.122 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:27:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:27:20.798+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 32911 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:27:21.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:21.255+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:21.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:21.255+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:21.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:21.255+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:21.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:21.256+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:21.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:21.122+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:21.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:21.122+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:21.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:21.122+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:21.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:21.122+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:21.760 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:21.501+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:21.760 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:21.501+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:21.760 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:21.501+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:21.760 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:21.501+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:22.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:22.217+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:22.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:22.217+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:22.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:22.217+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:22.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:22.218+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:22.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:22.074+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:22.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:22.074+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:22.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:22.074+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:22.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:22.075+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:22.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:22.523+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:22.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:22.523+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:22.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:22.523+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:22.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:22.524+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:23.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:23.262+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:23.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:23.262+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:23.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:23.262+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:23.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:23.263+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:23.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:23.056+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:23.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:23.056+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:23.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:23.056+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:23.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:23.057+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:23.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:23.537+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:23.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:23.537+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:23.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:23.537+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:23.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:23.538+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:24.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:27:24.161 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:27:24.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:24.281+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:24.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:24.281+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:24.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:24.281+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:24.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:24.282+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:24.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:24.090+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:24.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:24.090+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:24.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:24.090+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:24.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:24.091+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:24.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:24.573+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:24.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:24.573+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:24.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:24.573+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:24.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:24.574+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:25.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:25.305+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:25.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:25.305+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:25.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:25.305+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:25.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:25.305+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:25.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:25.130+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:25.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:25.130+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:25.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:25.130+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:25.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:25.131+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:25.813 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:27:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:27:25.802+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 32996 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:27:25.814 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:25.554+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:25.814 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:25.554+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:25.814 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:25.554+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:25.814 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:25.554+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:26.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:26.151+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:26.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:26.151+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:26.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:26.151+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:26.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:26.152+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:26.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:26.271+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:26.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:26.271+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:26.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:26.271+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:26.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:26.272+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:26.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:26.507+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:26.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:26.507+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:26.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:26.507+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:26.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:26.508+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:27.228 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:27:27.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:27.190+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:27.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:27.190+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:27.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:27.190+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:27.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:27.191+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:27.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:27.267+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:27.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:27.267+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:27.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:27.267+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:27.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:27.268+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:27.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:27.476+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:27.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:27.476+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:27.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:27.476+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:27.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:27.477+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:28.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:28.218+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:28.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:28.218+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:28.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:28.218+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:28.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:28.218+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:28.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:28.239+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:28.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:28.239+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:28.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:28.239+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:28.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:28.239+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:28.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:28.491+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:28.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:28.491+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:28.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:28.491+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:28.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:28.491+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:29.261 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:29.220+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:29.262 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:29.220+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:29.262 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:29.220+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:29.262 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:29.222+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:29.262 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:29.203+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:29.262 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:29.203+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:29.262 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:29.203+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:29.263 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:29.204+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:29.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:29.511+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:29.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:29.511+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:29.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:29.511+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:29.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:29.512+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:30.229 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:27:30.230 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:27:30.300 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:27:30.300 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-23T09:27:30.301 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-01-23T09:27:30.301 DEBUG:teuthology.orchestra.run.smithi059:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-23T09:27:30.316 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:27:30.317 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:27:30.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:30.167+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:30.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:30.167+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:30.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:30.167+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:30.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:30.168+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:30.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:30.206+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:30.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:30.206+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:30.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:30.206+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:30.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:30.207+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:30.806 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:30.535+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:30.806 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:30.535+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:30.807 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:30.535+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:30.807 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:30.536+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:31.178 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:27:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:27:30.806+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 33081 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:27:31.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:31.209+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:31.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:31.209+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:31.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:31.209+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:31.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:31.210+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:31.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:31.177+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:31.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:31.177+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:31.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:31.177+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:31.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:31.178+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:31.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:31.544+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:31.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:31.544+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:31.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:31.544+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:31.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:31.545+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:32.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:32.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:32.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:32.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:32.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:32.241+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:32.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:32.242+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:32.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:32.142+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:32.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:32.142+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:32.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:32.142+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:32.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:32.142+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:32.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:32.583+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:32.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:32.583+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:32.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:32.583+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:32.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:32.584+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:33.372 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:27:33.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:33.192+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:33.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:33.192+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:33.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:33.192+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:33.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:33.193+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:33.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:33.195+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:33.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:33.195+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:33.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:33.195+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:33.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:33.196+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:33.894 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:33.587+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:33.894 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:33.587+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:33.894 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:33.587+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:33.895 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:33.588+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:34.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:34.192+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:34.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:34.192+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:34.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:34.192+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:34.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:34.192+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:34.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:34.153+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:34.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:34.153+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:34.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:34.153+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:34.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:34.154+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:34.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:34.551+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:34.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:34.551+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:34.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:34.551+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:34.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:34.552+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:35.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:35.213+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:35.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:35.213+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:35.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:35.213+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:35.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:35.214+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:35.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:35.133+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:35.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:35.133+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:35.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:35.133+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:35.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:35.133+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:35.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:35.522+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:35.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:35.522+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:35.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:35.522+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:35.812 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:35.523+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:36.176 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:27:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:27:35.809+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 33166 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:27:36.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:36.259+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:36.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:36.259+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:36.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:36.259+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:36.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:36.260+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:36.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:36.176+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:36.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:36.176+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:36.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:36.176+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:36.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:36.177+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:36.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:36.481+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:36.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:36.481+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:36.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:36.481+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:36.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:36.482+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:37.468 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:37.300+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:37.468 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:37.300+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:37.469 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:37.300+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:37.469 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:37.301+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:37.469 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:37.175+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:37.469 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:37.175+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:37.469 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:37.175+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:37.469 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:37.176+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:37.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:37.468+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:37.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:37.468+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:37.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:37.468+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:37.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:37.469+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:38.462 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:38.311+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:38.462 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:38.311+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:38.462 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:38.311+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:38.462 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:38.312+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:38.463 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:38.184+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:38.463 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:38.184+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:38.463 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:38.184+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:38.463 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:38.184+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:38.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:38.462+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:38.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:38.462+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:38.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:38.462+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:38.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:38.462+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:39.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:39.305+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:39.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:39.305+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:39.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:39.305+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:39.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:39.306+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:39.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:39.178+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:39.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:39.178+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:39.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:39.178+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:39.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:39.179+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:39.751 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:39.509+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:39.751 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:39.509+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:39.751 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:39.509+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:39.752 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:39.510+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:40.270 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:40.187+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:40.270 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:40.187+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:40.270 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:40.187+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:40.271 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:40.188+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:40.522 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:40.270+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:40.522 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:40.270+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:40.522 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:40.270+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:40.522 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:40.271+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:40.814 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:40.533+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:40.814 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:40.533+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:40.814 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:40.533+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:40.814 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:40.534+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:41.218 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:27:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:27:40.813+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 33250 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:27:41.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:41.218+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:41.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:41.218+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:41.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:41.218+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:41.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:41.218+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:41.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:41.295+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:41.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:41.295+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:41.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:41.295+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:41.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:41.296+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:41.761 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:41.582+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:41.762 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:41.582+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:41.762 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:41.582+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:41.762 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:41.583+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:42.373 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:27:42.374 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:27:42.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:42.252+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:42.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:42.252+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:42.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:42.252+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:42.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:42.253+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:42.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:42.183+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:42.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:42.183+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:42.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:42.183+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:42.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:42.184+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:42.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:42.586+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:42.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:42.586+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:42.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:42.586+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:42.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:42.587+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:43.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:43.258+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:43.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:43.258+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:43.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:43.258+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:43.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:43.258+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:43.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:43.223+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:43.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:43.223+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:43.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:43.223+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:43.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:43.223+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:43.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:43.590+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:43.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:43.590+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:43.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:43.590+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:43.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:43.590+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:44.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:44.220+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:44.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:44.220+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:44.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:44.220+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:44.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:44.222+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:44.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:44.218+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:44.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:44.218+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:44.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:44.218+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:44.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:44.219+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:44.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:44.636+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:44.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:44.636+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:44.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:44.636+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:44.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:44.637+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:45.436 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:27:45.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:45.174+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:45.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:45.174+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:45.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:45.174+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:45.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:45.175+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:45.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:45.204+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:45.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:45.204+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:45.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:45.204+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:45.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:45.205+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:45.939 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:27:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:27:45.817+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 33335 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:27:45.939 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:45.610+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:45.939 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:45.610+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:45.939 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:45.610+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:45.940 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:45.610+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:46.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:46.138+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:46.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:46.138+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:46.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:46.138+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:46.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:46.139+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:46.591 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:46.231+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:46.591 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:46.231+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:46.591 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:46.231+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:46.591 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:46.232+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:46.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:46.590+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:46.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:46.590+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:46.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:46.590+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:46.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:46.591+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:47.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:47.166+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:47.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:47.166+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:47.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:47.166+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:47.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:47.167+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:47.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:47.263+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:47.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:47.263+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:47.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:47.263+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:47.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:47.264+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:47.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:47.595+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:47.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:47.595+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:47.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:47.595+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:47.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:47.596+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:48.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:48.134+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:48.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:48.134+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:48.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:48.134+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:48.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:48.135+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:48.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:48.277+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:48.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:48.277+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:48.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:48.277+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:48.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:48.279+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:48.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:48.556+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:48.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:48.556+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:48.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:48.557+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:48.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:48.557+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:49.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:49.164+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:49.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:49.164+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:49.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:49.164+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:49.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:49.165+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:49.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:49.236+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:49.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:49.236+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:49.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:49.236+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:49.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:49.237+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:49.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:49.529+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:49.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:49.529+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:49.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:49.529+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:49.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:49.530+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:50.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:50.122+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:50.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:50.123+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:50.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:50.123+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:50.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:50.124+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:50.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:50.221+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:50.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:50.221+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:50.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:50.221+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:50.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:50.222+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:50.822 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:50.563+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:50.822 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:50.563+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:50.822 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:50.563+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:50.823 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:50.564+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:51.116 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:27:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:27:50.821+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 33419 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:27:51.366 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:51.115+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:51.366 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:51.115+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:51.367 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:51.115+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:51.367 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:51.116+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:51.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:51.267+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:51.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:51.267+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:51.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:51.267+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:51.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:51.267+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:51.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:51.578+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:51.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:51.578+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:51.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:51.578+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:51.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:51.579+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:52.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:52.112+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:52.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:52.112+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:52.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:52.112+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:52.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:52.113+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:52.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:52.276+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:52.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:52.276+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:52.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:52.276+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:52.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:52.277+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:52.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:52.617+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:52.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:52.617+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:52.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:52.617+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:52.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:52.618+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:53.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:53.138+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:53.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:53.138+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:53.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:53.138+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:53.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:53.139+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:53.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:53.251+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:53.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:53.251+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:53.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:53.251+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:53.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:53.252+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:53.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:53.570+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:53.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:53.570+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:53.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:53.570+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:53.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:53.570+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:54.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:54.100+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:54.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:54.100+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:54.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:54.100+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:54.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:54.100+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:54.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:54.261+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:54.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:54.261+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:54.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:54.261+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:54.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:54.262+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:54.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:54.558+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:54.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:54.558+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:54.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:54.558+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:54.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:54.559+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:55.398 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:55.122+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:55.398 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:55.122+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:55.399 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:55.122+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:55.399 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:55.123+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:55.399 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:55.221+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:55.399 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:55.221+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:55.399 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:55.221+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:55.399 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:55.222+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:55.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:55.514+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:55.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:55.514+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:55.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:55.514+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:55.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:55.515+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:56.128 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:27:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:27:55.825+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 33504 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:27:56.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:56.127+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:56.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:56.127+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:56.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:56.127+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:56.470 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:56.128+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:56.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:56.190+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:56.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:56.190+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:56.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:56.190+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:56.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:56.190+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:56.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:56.470+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:56.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:56.470+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:56.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:56.470+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:56.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:56.471+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:57.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:27:57.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:27:57.443 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:57.185+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:57.444 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:57.185+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:57.444 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:57.185+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:57.444 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:57.185+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:57.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:57.175+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:57.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:57.175+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:57.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:57.175+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:57.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:57.176+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:57.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:57.443+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:57.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:57.443+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:57.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:57.443+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:57.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:57.444+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:58.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:58.140+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:58.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:58.140+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:58.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:58.140+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:58.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:58.140+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:58.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:58.162+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:58.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:58.162+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:58.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:58.162+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:58.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:58.162+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:58.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:58.491+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:58.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:58.491+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:58.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:58.491+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:58.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:58.492+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:59.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:59.124+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:59.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:59.124+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:59.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:59.124+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:27:59.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:27:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:27:59.124+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:27:59.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:59.162+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:59.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:59.162+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:59.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:59.162+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:27:59.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:27:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:27:59.162+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:27:59.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:59.450+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:59.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:59.450+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:59.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:59.450+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:27:59.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:27:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:27:59.451+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:00.460 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:00.166+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:00.461 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:00.166+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:00.461 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:00.166+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:00.461 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:00.167+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:00.461 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:00.082+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:00.461 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:00.082+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:00.461 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:00.082+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:00.461 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:00.083+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:00.508 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:28:00.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:00.460+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:00.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:00.460+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:00.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:00.460+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:00.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:00.461+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:01.080 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:28:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:28:00.829+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 33588 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:28:01.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:01.080+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:01.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:01.080+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:01.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:01.080+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:01.455 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:01.081+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:01.455 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:01.196+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:01.456 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:01.196+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:01.456 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:01.196+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:01.456 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:01.196+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:01.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:01.454+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:01.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:01.454+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:01.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:01.454+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:01.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:01.455+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:02.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:02.077+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:02.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:02.077+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:02.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:02.077+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:02.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:02.078+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:02.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:02.185+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:02.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:02.185+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:02.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:02.185+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:02.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:02.186+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:02.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:02.475+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:02.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:02.475+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:02.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:02.475+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:02.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:02.476+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:03.436 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:03.166+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:03.437 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:03.166+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:03.437 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:03.166+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:03.437 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:03.166+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:03.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:03.082+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:03.437 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:03.082+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:03.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:03.082+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:03.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:03.083+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:03.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:28:03.510 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:28:03.584 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:28:03.584 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-23T09:28:03.585 INFO:tasks.cephadm:Archiving crash dumps... 2024-01-23T09:28:03.587 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/crash to /home/teuthworker/archive/cbodley-2024-01-22_19:46:51-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7527396/remote/smithi059/crash 2024-01-23T09:28:03.587 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/crash -- . 2024-01-23T09:28:03.656 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/crash to /home/teuthworker/archive/cbodley-2024-01-22_19:46:51-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7527396/remote/smithi137/crash 2024-01-23T09:28:03.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:28:03.658 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:28:03.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:03.436+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:03.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:03.436+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:03.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:03.436+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:03.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:03.437+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:04.410 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:04.047+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:04.411 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:04.047+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:04.411 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:04.047+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:04.411 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:04.048+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:04.411 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:04.199+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:04.412 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:04.199+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:04.412 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:04.199+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:04.412 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:04.199+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:04.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:04.410+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:04.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:04.410+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:04.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:04.410+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:04.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:04.411+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:05.431 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:05.187+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:05.431 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:05.187+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:05.431 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:05.187+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:05.431 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:05.188+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:05.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:05.038+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:05.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:05.038+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:05.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:05.038+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:05.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:05.039+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:05.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:05.430+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:05.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:05.430+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:05.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:05.430+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:05.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:05.431+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:06.084 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:28:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:28:05.833+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 33674 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:28:06.402 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:06.084+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:06.402 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:06.084+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:06.402 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:06.084+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:06.403 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:06.085+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:06.403 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:06.140+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:06.403 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:06.140+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:06.403 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:06.140+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:06.403 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:06.141+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:06.716 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:28:06.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:06.402+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:06.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:06.402+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:06.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:06.402+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:06.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:06.403+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:07.422 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:07.049+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:07.422 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:07.049+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:07.423 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:07.049+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:07.423 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:07.049+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:07.423 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:07.172+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:07.423 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:07.172+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:07.423 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:07.172+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:07.423 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:07.172+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:07.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:07.421+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:07.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:07.421+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:07.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:07.421+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:07.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:07.422+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:08.375 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:08.142+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:08.375 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:08.142+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:08.375 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:08.142+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:08.375 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:08.142+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:08.375 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:08.069+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:08.376 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:08.069+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:08.376 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:08.069+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:08.376 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:08.070+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:08.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:08.373+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:08.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:08.373+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:08.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:08.373+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:08.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:08.374+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:09.344 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:09.025+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:09.344 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:09.025+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:09.344 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:09.025+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:09.344 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:09.026+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:09.345 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:09.178+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:09.345 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:09.178+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:09.345 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:09.178+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:09.345 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:09.179+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:09.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:09.342+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:09.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:09.342+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:09.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:09.342+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:09.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:09.343+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:10.340 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:10.132+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:10.341 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:10.132+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:10.341 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:10.132+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:10.341 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:10.133+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:10.341 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:10.070+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:10.341 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:10.070+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:10.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:10.070+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:10.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:10.070+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:10.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:10.339+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:10.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:10.339+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:10.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:10.339+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:10.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:10.340+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:11.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:28:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:28:10.836+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 33759 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:28:11.382 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:11.150+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:11.382 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:11.150+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:11.382 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:11.150+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:11.382 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:11.151+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:11.383 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:11.045+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:11.383 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:11.045+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:11.383 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:11.045+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:11.383 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:11.046+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:11.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:11.380+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:11.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:11.380+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:11.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:11.380+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:11.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:11.381+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:12.134 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:12.004+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:12.134 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:12.004+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:12.134 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:12.004+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:12.134 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:12.005+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:12.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:12.133+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:12.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:12.133+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:12.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:12.133+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:12.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:12.134+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:12.477 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:12.358+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:12.477 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:12.358+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:12.477 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:12.358+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:12.477 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:12.358+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:13.365 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:13.030+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:13.365 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:13.030+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:13.365 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:13.030+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:13.366 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:13.031+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:13.366 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:13.156+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:13.366 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:13.156+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:13.366 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:13.156+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:13.366 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:13.156+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:13.640 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:13.364+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:13.640 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:13.364+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:13.640 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:13.364+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:13.640 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:13.364+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:14.395 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:14.060+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:14.395 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:14.060+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:14.395 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:14.060+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:14.396 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:14.061+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:14.396 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:14.195+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:14.396 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:14.195+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:14.396 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:14.195+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:14.396 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:14.196+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:14.727 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:14.394+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:14.728 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:14.394+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:14.728 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:14.394+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:14.728 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:14.394+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:15.394 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:15.154+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:15.395 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:15.154+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:15.395 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:15.154+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:15.395 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:15.155+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:15.395 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:15.071+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:15.395 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:15.071+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:15.395 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:15.071+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:15.396 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:15.073+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:15.717 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:28:15.718 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:28:15.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:15.393+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:15.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:15.393+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:15.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:15.393+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:15.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:15.394+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:16.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:28:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:28:15.840+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 33845 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:28:16.141 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:16.039+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:16.141 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:16.039+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:16.141 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:16.039+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:16.141 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:16.040+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:16.393 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:16.139+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:16.393 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:16.139+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:16.393 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:16.139+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:16.393 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:16.140+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:16.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:16.392+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:16.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:16.392+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:16.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:16.392+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:16.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:16.392+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:17.380 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:17.159+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:17.381 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:17.159+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:17.381 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:17.159+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:17.381 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:17.160+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:17.381 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:17.040+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:17.381 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:17.040+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:17.381 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:17.040+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:17.382 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:17.040+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:17.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:17.379+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:17.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:17.379+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:17.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:17.379+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:17.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:17.380+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:18.429 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:18.051+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:18.429 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:18.051+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:18.429 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:18.051+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:18.429 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:18.052+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:18.429 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:18.185+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:18.430 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:18.185+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:18.430 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:18.185+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:18.430 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:18.186+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:18.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:18.427+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:18.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:18.427+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:18.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:18.427+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:18.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:18.428+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:18.784 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:28:19.414 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:19.073+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:19.414 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:19.073+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:19.414 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:19.073+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:19.414 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:19.074+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:19.415 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:19.189+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:19.415 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:19.189+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:19.415 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:19.189+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:19.415 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:19.189+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:19.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:19.412+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:19.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:19.412+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:19.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:19.412+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:19.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:19.413+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:20.396 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:20.211+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:20.397 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:20.211+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:20.397 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:20.211+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:20.397 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:20.212+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:20.397 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:20.081+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:20.397 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:20.081+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:20.398 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:20.081+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:20.398 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:20.082+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:20.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:20.395+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:20.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:20.395+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:20.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:20.395+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:20.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:20.396+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:21.117 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:28:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:28:20.844+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 33929 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:28:21.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:21.116+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:21.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:21.116+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:21.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:21.116+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:21.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:21.117+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:21.447 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:21.232+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:21.447 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:21.232+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:21.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:21.232+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:21.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:21.233+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:21.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:21.445+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:21.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:21.445+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:21.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:21.445+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:21.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:21.446+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:22.439 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:22.162+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:22.439 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:22.162+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:22.440 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:22.162+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:22.440 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:22.163+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:22.440 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:22.248+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:22.440 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:22.248+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:22.440 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:22.248+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:22.440 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:22.248+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:22.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:22.438+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:22.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:22.438+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:22.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:22.438+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:22.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:22.439+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:23.465 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:23.204+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:23.465 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:23.204+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:23.466 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:23.204+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:23.466 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:23.205+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:23.466 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:23.258+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:23.466 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:23.258+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:23.466 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:23.258+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:23.466 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:23.259+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:23.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:23.464+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:23.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:23.464+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:23.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:23.464+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:23.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:23.464+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:24.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:24.203+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:24.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:24.203+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:24.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:24.203+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:24.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:24.203+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:24.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:24.435+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:24.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:24.435+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:24.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:24.435+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:24.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:24.436+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:24.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:24.304+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:24.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:24.304+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:24.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:24.304+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:24.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:24.305+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:25.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:25.227+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:25.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:25.227+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:25.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:25.227+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:25.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:25.228+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:25.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:25.449+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:25.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:25.449+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:25.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:25.449+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:25.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:25.450+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:25.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:25.301+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:25.726 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:25.301+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:25.726 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:25.301+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:25.726 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:25.301+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:26.118 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:28:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:28:25.848+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 34015 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:28:26.424 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:26.239+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:26.425 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:26.239+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:26.425 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:26.239+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:26.425 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:26.239+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:26.425 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:26.301+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:26.425 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:26.301+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:26.426 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:26.301+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:26.426 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:26.301+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:26.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:26.423+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:26.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:26.423+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:26.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:26.423+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:26.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:26.424+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:27.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:27.253+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:27.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:27.253+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:27.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:27.253+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:27.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:27.254+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:27.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:27.321+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:27.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:27.321+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:27.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:27.321+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:27.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:27.321+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:27.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:27.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:27.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:27.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:27.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:27.376+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:27.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:27.376+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:28.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:28.277+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:28.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:28.277+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:28.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:28.277+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:28.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:28.277+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:28.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:28.399+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:28.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:28.399+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:28.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:28.399+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:28.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:28.400+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:28.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:28.215+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:28.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:28.215+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:28.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:28.215+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:28.476 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:28.215+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:29.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:29.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:29.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:29.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:29.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:29.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:29.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:29.198+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:29.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:29.234+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:29.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:29.234+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:29.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:29.234+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:29.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:29.234+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:29.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:29.379+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:29.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:29.379+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:29.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:29.379+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:29.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:29.380+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:30.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:30.212+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:30.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:30.212+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:30.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:30.212+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:30.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:30.213+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:30.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:30.253+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:30.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:30.253+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:30.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:30.253+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:30.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:30.254+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:30.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:30.357+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:30.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:30.358+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:30.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:30.358+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:30.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:30.358+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:30.785 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:28:30.785 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:28:31.224 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:28:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:28:30.853+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 34099 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:28:31.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:31.296+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:31.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:31.296+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:31.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:31.296+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:31.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:31.296+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:31.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:31.237+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:31.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:31.237+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:31.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:31.237+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:31.725 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:31.238+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:31.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:31.358+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:31.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:31.358+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:31.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:31.358+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:31.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:31.359+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:32.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:32.192+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:32.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:32.192+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:32.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:32.192+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:32.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:32.193+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:32.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:32.250+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:32.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:32.250+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:32.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:32.250+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:32.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:32.251+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:32.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:32.367+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:32.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:32.367+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:32.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:32.367+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:32.476 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:32.367+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:33.412 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:33.294+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:33.413 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:33.295+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:33.413 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:33.295+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:33.413 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:33.295+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:33.413 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:33.153+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:33.413 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:33.153+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:33.413 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:33.153+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:33.414 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:33.153+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:33.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:33.412+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:33.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:33.412+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:33.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:33.412+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:33.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:33.413+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:33.852 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:28:34.399 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:34.111+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:34.400 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:34.111+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:34.400 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:34.111+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:34.400 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:34.112+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:34.400 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:34.253+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:34.400 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:34.253+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:34.401 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:34.253+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:34.401 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:34.253+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:34.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:34.399+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:34.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:34.399+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:34.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:34.399+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:34.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:34.400+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:35.443 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:35.105+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:35.443 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:35.105+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:35.443 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:35.105+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:35.443 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:35.106+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:35.443 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:35.258+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:35.443 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:35.258+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:35.444 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:35.258+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:35.444 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:35.258+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:35.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:35.442+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:35.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:35.442+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:35.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:35.442+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:35.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:35.443+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:36.148 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:28:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:28:35.857+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 34185 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:28:36.417 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:36.148+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:36.417 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:36.148+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:36.417 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:36.148+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:36.417 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:36.148+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:36.418 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:36.271+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:36.418 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:36.271+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:36.418 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:36.271+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:36.418 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:36.272+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:36.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:36.417+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:36.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:36.417+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:36.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:36.417+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:36.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:36.417+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:36.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:28:36.854 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:28:36.924 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:28:36.924 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-23T09:28:36.925 INFO:tasks.cephadm:Checking cluster log for badness... 2024-01-23T09:28:36.925 DEBUG:teuthology.orchestra.run.smithi059:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-01-23T09:28:36.958 INFO:teuthology.orchestra.run.smithi059.stderr:grep: /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph.log: No such file or directory 2024-01-23T09:28:36.960 INFO:tasks.cephadm:Compressing logs... 2024-01-23T09:28:36.960 DEBUG:teuthology.orchestra.run.smithi059:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-23T09:28:37.002 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:28:37.003 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:28:37.026 INFO:teuthology.orchestra.run.smithi059.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-23T09:28:37.027 INFO:teuthology.orchestra.run.smithi059.stderr:: No such file or directory 2024-01-23T09:28:37.027 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-mds.cephfs.smithi059.lsxsmm.log 2024-01-23T09:28:37.027 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-mds.cephfs.smithi059.twpxfz.log 2024-01-23T09:28:37.028 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-mds.cephfs.smithi059.lsxsmm.log: gzip -5 --verbose -- /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-osd.1.log 2024-01-23T09:28:37.029 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-mds.cephfs.smithi059.twpxfz.log: gzip -5 --verbose -- /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-mgr.smithi059.bzqirf.log 2024-01-23T09:28:37.029 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-osd.0.log 2024-01-23T09:28:37.030 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-osd.1.log: /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-mgr.smithi059.bzqirf.log: /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-osd.2.log 2024-01-23T09:28:37.030 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-volume.log 2024-01-23T09:28:37.033 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-osd.2.log: /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-mon.smithi059.log 2024-01-23T09:28:37.035 INFO:teuthology.orchestra.run.smithi059.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-23T09:28:37.047 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-mon.smithi059.log: 90.7% -- replaced with /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-volume.log.gz 2024-01-23T09:28:37.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:37.115+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:37.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:37.115+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:37.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:37.115+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:37.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:37.115+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:37.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:37.309+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:37.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:37.309+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:37.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:37.309+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:37.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:37.310+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:37.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:37.397+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:37.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:37.397+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:37.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:37.397+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:37.726 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:37.398+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:37.918 INFO:teuthology.orchestra.run.smithi059.stderr:gzip: /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-mgr.smithi059.bzqirf.log: file size changed while zipping 2024-01-23T09:28:37.918 INFO:teuthology.orchestra.run.smithi059.stderr: 89.9% -- replaced with /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-mgr.smithi059.bzqirf.log.gz 2024-01-23T09:28:38.437 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:38.326+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:38.437 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:38.326+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:38.437 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:38.326+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:38.437 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:38.327+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:38.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:38.148+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:38.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:38.148+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:38.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:38.148+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:38.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:38.148+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:38.438 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:38.435+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:38.439 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:38.435+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:38.439 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:38.435+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:38.439 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:38.436+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:38.473 INFO:teuthology.orchestra.run.smithi059.stderr:gzip: /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-mon.smithi059.log: file size changed while zipping 2024-01-23T09:28:38.474 INFO:teuthology.orchestra.run.smithi059.stderr: 92.8% -- replaced with /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-mon.smithi059.log.gz 2024-01-23T09:28:39.416 INFO:teuthology.orchestra.run.smithi059.stderr: 95.3% -- replaced with /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-mds.cephfs.smithi059.lsxsmm.log.gz 2024-01-23T09:28:39.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:39.142+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:39.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:39.142+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:39.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:39.142+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:39.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:39.143+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:39.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:39.343+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:39.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:39.343+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:39.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:39.343+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:39.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:39.343+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:39.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:39.483+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:39.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:39.484+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:39.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:39.484+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:39.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:39.484+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:40.060 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:28:40.265 INFO:teuthology.orchestra.run.smithi059.stderr:gzip: /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-osd.2.log: file size changed while zipping 2024-01-23T09:28:40.265 INFO:teuthology.orchestra.run.smithi059.stderr: 91.3% -- replaced with /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-osd.2.log.gz 2024-01-23T09:28:40.464 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:40.371+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:40.464 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:40.371+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:40.464 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:40.371+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:40.464 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:40.372+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:40.465 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:40.145+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:40.465 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:40.145+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:40.465 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:40.145+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:40.465 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:40.146+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:40.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:40.463+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:40.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:40.463+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:40.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:40.463+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:40.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:40.464+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:41.224 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:28:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:28:40.861+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 34269 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:28:41.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:41.104+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:41.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:41.104+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:41.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:41.104+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:41.225 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:41.105+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:41.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:41.355+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:41.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:41.355+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:41.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:41.355+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:41.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:41.356+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:41.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:41.436+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:41.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:41.436+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:41.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:41.436+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:41.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:41.437+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:42.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:42.139+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:42.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:42.139+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:42.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:42.139+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:42.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:42.140+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:42.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:42.356+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:42.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:42.356+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:42.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:42.356+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:42.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:42.357+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:42.684 INFO:teuthology.orchestra.run.smithi059.stderr:gzip: /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-osd.0.log: file size changed while zipping 2024-01-23T09:28:42.684 INFO:teuthology.orchestra.run.smithi059.stderr: 95.0% -- replaced with /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-osd.0.log.gz 2024-01-23T09:28:42.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:42.449+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:42.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:42.449+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:42.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:42.449+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:42.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:42 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:42.450+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:43.470 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:43.367+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:43.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:43.367+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:43.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:43.367+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:43.471 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:43.368+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:43.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:43.171+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:43.471 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:43.171+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:43.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:43.171+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:43.472 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:43.172+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:43.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:43.470+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:43.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:43.470+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:43.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:43.470+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:43.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:43 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:43.470+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:44.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:44.357+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:44.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:44.357+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:44.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:44.357+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:44.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:44.358+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:44.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:44.160+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:44.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:44.160+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:44.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:44.160+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:44.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:44.161+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:44.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:44.504+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:44.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:44.504+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:44.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:44.504+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:44.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:44 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:44.504+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:45.218 INFO:teuthology.orchestra.run.smithi059.stderr:gzip: /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-osd.1.log: file size changed while zipping 2024-01-23T09:28:45.218 INFO:teuthology.orchestra.run.smithi059.stderr: 96.4% -- replaced with /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-osd.1.log.gz 2024-01-23T09:28:45.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:45.116+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:45.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:45.116+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:45.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:45.116+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:45.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:45.117+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:45.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:45.332+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:45.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:45.332+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:45.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:45.332+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:45.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:45.333+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:45.866 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:45.503+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:45.866 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:45.503+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:45.866 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:45.503+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:45.866 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:45.504+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:46.132 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:28:45 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:28:45.865+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 34355 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:28:46.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:46.326+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:46.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:46.326+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:46.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:46.326+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:46.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:46.327+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:46.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:46.132+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:46.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:46.132+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:46.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:46.132+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:46.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:46.133+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:46.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:46.548+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:46.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:46.548+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:46.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:46.548+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:46.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:46 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:46.549+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:47.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:47.351+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:47.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:47.351+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:47.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:47.351+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:47.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:47.352+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:47.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:47.129+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:47.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:47.129+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:47.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:47.129+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:47.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:47.129+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:47.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:47.516+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:47.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:47.516+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:47.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:47.516+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:47.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:47 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:47.517+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:48.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:48.173+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:48.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:48.173+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:48.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:48.173+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:48.473 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:48.174+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:48.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:48.392+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:48.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:48.392+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:48.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:48.392+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:48.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:48.392+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:48.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:48.472+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:48.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:48.472+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:48.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:48.472+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:48.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:48 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:48.473+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:49.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:28:49.062 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:28:49.378 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:49.126+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:49.378 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:49.126+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:49.378 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:49.126+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:49.378 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:49.127+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:49.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:49.377+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:49.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:49.377+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:49.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:49.377+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:49.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:49.378+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:49.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:49.487+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:49.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:49.487+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:49.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:49.487+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:49.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:49 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:49.487+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:50.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:50.374+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:50.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:50.374+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:50.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:50.374+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:50.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:50.375+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:50.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:50.173+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:50.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:50.173+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:50.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:50.173+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:50.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:50.174+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:50.870 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:50.484+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:50.870 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:50.484+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:50.870 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:50.484+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:50.871 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:50.485+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:51.141 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:28:50 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:28:50.869+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 34439 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:28:51.394 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:51.141+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:51.394 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:51.141+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:51.394 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:51.141+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:51.394 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:51.141+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:51.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:51.498+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:51.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:51.498+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:51.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:51.498+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:51.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:51.498+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:51.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:51.394+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:51.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:51.394+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:51.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:51.394+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:51.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:51 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:51.395+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:52.124 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:28:52.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:52.369+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:52.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:52.369+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:52.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:52.369+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:52.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:52.370+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:52.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:52.131+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:52.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:52.131+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:52.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:52.131+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:52.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:52.132+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:52.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:52.537+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:52.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:52.537+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:52.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:52.537+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:52.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:52 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:52.537+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:53.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:53.173+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:53.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:53.173+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:53.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:53.173+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:53.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:53.174+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:53.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:53.376+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:53.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:53.376+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:53.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:53.376+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:53.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:53.377+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:53.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:53.587+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:53.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:53.587+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:53.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:53.587+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:53.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:53 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:53.588+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:54.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:54.156+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:54.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:54.156+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:54.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:54.156+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:54.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:54.157+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:54.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:54.381+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:54.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:54.381+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:54.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:54.381+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:54.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:54.381+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:54.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:54.541+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:54.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:54.541+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:54.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:54.541+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:54.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:54 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:54.542+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:55.376 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:55.119+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:55.376 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:55.119+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:55.376 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:55.119+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:55.376 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:55.119+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:55.681 INFO:teuthology.orchestra.run.smithi059.stderr: 92.2% -- replaced with /var/log/ceph/3fc07a38-b9cb-11ee-95b1-87774f69a715/ceph-mds.cephfs.smithi059.twpxfz.log.gz 2024-01-23T09:28:55.682 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:55.375+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:55.682 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:55.375+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:55.682 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:55.375+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:55.682 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:55.376+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:55.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:55.559+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:55.682 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:55.559+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:55.683 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:55.559+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:55.683 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:55.560+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:55.684 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-01-23T09:28:55.684 INFO:teuthology.orchestra.run.smithi059.stderr:real 0m18.670s 2024-01-23T09:28:55.684 INFO:teuthology.orchestra.run.smithi059.stderr:user 0m39.416s 2024-01-23T09:28:55.684 INFO:teuthology.orchestra.run.smithi059.stderr:sys 0m0.891s 2024-01-23T09:28:55.974 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:28:55 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:28:55.873+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 34525 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:28:56.347 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:56.104+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:56.348 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:56.104+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:56.348 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:56.104+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:56.348 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:56.105+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:56.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:56.347+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:56.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:56.347+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:56.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:56.347+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:56.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:56.348+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:56.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:56.537+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:56.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:56.537+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:56.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:56.537+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:56.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:56 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:56.538+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:57.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:57.151+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:57.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:57.151+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:57.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:57.151+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:57.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:57.152+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:57.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:57.344+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:57.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:57.344+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:57.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:57.344+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:57.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:57.345+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:57.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:57.547+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:57.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:57.547+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:57.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:57.547+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:57.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:57 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:57.548+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:58.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:58.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:58.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:58.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:58.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:58.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:58.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:58.198+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:58.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:58.369+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:58.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:58.369+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:58.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:58.369+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:58.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:58.370+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:58.977 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:58.591+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:58.977 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:58.591+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:58.977 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:58.591+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:58.977 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:58 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:58.592+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:59.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:59.167+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:59.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:59.167+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:59.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:59.167+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:28:59.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:28:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:28:59.167+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:28:59.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:59.395+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:59.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:59.395+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:59.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:59.395+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:28:59.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:28:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:28:59.396+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:28:59.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:59.637+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:59.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:59.637+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:59.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:59.637+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:28:59.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:28:59 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:28:59.638+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:00.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:00.180+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:00.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:00.180+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:00.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:00.180+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:00.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:00.181+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:00.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:00.358+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:00.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:00.358+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:00.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:00.358+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:00.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:00.359+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:00.889 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:29:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:29:00.877+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 34608 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:29:00.889 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:00.630+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:00.889 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:00.630+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:00.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:00.630+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:00.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:00 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:00.630+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:01.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:01.206+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:01.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:01.206+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:01.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:01.206+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:01.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:01.207+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:01.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:01.329+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:01.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:01.329+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:01.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:01.329+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:01.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:01.330+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:01.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:01.605+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:01.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:01.605+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:01.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:01.605+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:01.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:01 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:01.605+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:02.656 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:02.243+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:02.656 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:02.243+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:02.656 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:02.243+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:02.656 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:02.243+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:02.656 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:02.293+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:02.657 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:02.293+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:02.657 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:02.293+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:02.657 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:02.293+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:02.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:02.655+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:02.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:02.655+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:02.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:02.655+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:02.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:02 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:02.656+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:03.643 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:03.287+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:03.643 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:03.287+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:03.643 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:03.287+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:03.643 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:03.288+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:03.644 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:03.287+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:03.644 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:03.287+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:03.644 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:03.287+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:03.644 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:03.288+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:03.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:03.681+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:03.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:03.681+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:03.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:03.681+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:03.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:03 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:03.681+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:04.125 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:29:04.126 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:29:04.666 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:04.262+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:04.667 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:04.262+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:04.667 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:04.262+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:04.667 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:04.263+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:04.667 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:04.277+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:04.667 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:04.277+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:04.667 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:04.277+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:04.667 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:04.277+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:04.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:04.666+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:04.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:04.666+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:04.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:04.666+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:04.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:04 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:04.667+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:05.634 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:05.278+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:05.634 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:05.278+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:05.634 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:05.278+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:05.634 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:05.279+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:05.635 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:05.241+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:05.635 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:05.241+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:05.635 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:05.241+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:05.635 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:05.242+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:05.894 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:29:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:29:05.881+0000 7f5c9756c700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 34694 slow ops, oldest is log(1 entries from seq 289 at 2024-01-23T08:51:52.982568+0000) 2024-01-23T09:29:05.895 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:05.633+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:05.895 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:05.633+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:05.895 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:05.633+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:05.895 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:05 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:05.634+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:06.639 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:06.323+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:06.640 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:06.323+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:06.640 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:06.323+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:06.640 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:06.323+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:06.640 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:06.275+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:06.641 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:06.275+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:06.641 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:06.275+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:06.641 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:06.275+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:06.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:06.639+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:06.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:06.639+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:06.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:06.639+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:06.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:06 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:06.639+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:07.200 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:29:07.670 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:07.329+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:07.670 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:07.329+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:07.670 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:07.329+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:07.670 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:07.330+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:07.671 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:07.319+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:07.671 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:07.319+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:07.671 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:07.319+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:07.671 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:07.319+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:07.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:07.669+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:07.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:07.669+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:07.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:07.669+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:07.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:07 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:07.670+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:08.710 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:08.306+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:08.711 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:08.306+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:08.711 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:08.306+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:08.711 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:08.307+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:08.711 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:08.366+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:08.711 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:08.366+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:08.711 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:08.366+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:08.712 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:08.366+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:08.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:08.710+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:08.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:08.710+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:08.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:08.710+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:08.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:08 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:08.710+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:09.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:09.321+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:09.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:09.321+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:09.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:09.321+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:09.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:09.322+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:09.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:09.309+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:09.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:09.309+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:09.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:09.309+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:09.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:09.310+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:09.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:09.697+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:09.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:09.697+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:09.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:09.697+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:09.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:09 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:09.697+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:10.201 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:29:10.202 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:29:10.268 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:29:10.268 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-23T09:29:10.268 INFO:tasks.cephadm:Removing cluster... 2024-01-23T09:29:10.269 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 --force 2024-01-23T09:29:10.557 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:10.326+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:10.557 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:10.326+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:10.558 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:10.326+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:10.558 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:10.327+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:10.558 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:10.294+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:10.558 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:10.294+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:10.558 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:10.294+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:10.559 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:10.295+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:10.815 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:29:10 smithi059 systemd[1]: Stopping Ceph mon.smithi059 for 3fc07a38-b9cb-11ee-95b1-87774f69a715... 2024-01-23T09:29:10.815 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:10.743+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:10.816 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:10.743+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:10.816 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:10.743+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:10.816 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:10.744+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:11.180 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:29:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:29:10.814+0000 7f5c9a572700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi059 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-01-23T09:29:11.180 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:29:10 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059[28420]: 2024-01-23T09:29:10.814+0000 7f5c9a572700 -1 mon.smithi059@0(probing) e2 *** Got Signal Terminated *** 2024-01-23T09:29:11.474 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:29:11 smithi059 podman[84509]: 2024-01-23 09:29:11.180741027 +0000 UTC m=+0.474434721 container died 810a40012e10c7bcb0c50786b86e91f99951825a767752b4654b71ec2df613f0 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240118, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.31.3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=quincy-b824e5b) 2024-01-23T09:29:11.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:11.303+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:11.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:11.303+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:11.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:11.303+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:11.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:11.304+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:11.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:11.323+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:11.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:11.323+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:11.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:11.323+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:11.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:11.324+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:11.767 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:29:11 smithi059 podman[84509]: 2024-01-23 09:29:11.582708434 +0000 UTC m=+0.876402134 container cleanup 810a40012e10c7bcb0c50786b86e91f99951825a767752b4654b71ec2df613f0 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059, ceph=True, GIT_CLEAN=True, io.buildah.version=1.31.3, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=quincy-b824e5b, org.label-schema.build-date=20240118, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-01-23T09:29:11.767 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:29:11 smithi059 bash[84509]: ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059 2024-01-23T09:29:11.767 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:11.730+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:11.767 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:11.730+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:11.767 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:11.730+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:11.767 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:11 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:11.731+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:12.024 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:29:11 smithi059 podman[84523]: 2024-01-23 09:29:11.84281066 +0000 UTC m=+0.667367919 container remove 810a40012e10c7bcb0c50786b86e91f99951825a767752b4654b71ec2df613f0 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-mon-smithi059, RELEASE=quincy-b824e5b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240118, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.31.3, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-01-23T09:29:12.024 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:29:12 smithi059 systemd[1]: ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@mon.smithi059.service: Deactivated successfully. 2024-01-23T09:29:12.024 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:29:12 smithi059 systemd[1]: Stopped Ceph mon.smithi059 for 3fc07a38-b9cb-11ee-95b1-87774f69a715. 2024-01-23T09:29:12.024 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Jan 23 09:29:12 smithi059 systemd[1]: ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@mon.smithi059.service: Consumed 23.850s CPU time. 2024-01-23T09:29:12.369 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:12.268+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:12.370 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:12.268+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:12.370 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:12.268+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:12.370 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:12.269+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:12.682 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:12.369+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:12.682 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:12.369+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:12.682 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:12.369+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:12.682 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:12.370+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:12.955 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:12.681+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:12.955 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:12.681+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:12.955 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:12.681+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:12.956 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:12 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:12.682+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:13.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:13.308+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:13.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:13.308+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:13.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:13.308+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:13.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:13.309+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:13.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:13.419+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:13.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:13.419+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:13.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:13.419+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:13.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:13.420+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:13.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:13.646+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:13.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:13.646+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:13.811 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:13.646+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:13.812 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:13 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:13.647+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:14.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:14.321+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:14.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:14.321+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:14.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:14.321+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:14.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:14.322+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:14.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:14.604+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:14.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:14.604+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:14.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:14.604+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:14.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:14.605+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:14.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:14.452+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:14.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:14.452+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:14.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:14.452+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:14.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:14 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:14.453+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:15.409 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:15.317+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:15.409 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:15.317+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:15.409 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:15.317+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:15.409 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:15.317+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:15.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:15.436+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:15.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:15.436+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:15.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:15.436+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:15.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:15.436+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:15.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:15.603+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:15.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:15.603+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:15.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:15.603+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:15.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:15 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:15.604+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:16.462 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:16.391+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:16.462 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:16.391+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:16.463 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:16.391+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:16.463 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:16.391+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:16.463 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:16.335+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:16.463 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:16.335+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:16.463 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:16.335+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:16.463 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:16.336+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:16.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:16.643+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:16.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:16.643+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:16.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:16.643+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:16.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:16 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:16.644+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:17.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:17.296+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:17.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:17.296+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:17.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:17.296+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:17.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:17.298+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:17.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:17.411+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:17.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:17.411+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:17.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:17.411+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:17.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:17.412+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:17.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:17.611+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:17.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:17.611+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:17.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:17.611+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:17.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:17 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:17.612+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:18.571 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:18.402+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:18.571 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:18.402+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:18.572 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:18.402+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:18.572 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:18.403+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:18.572 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:18.280+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:18.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:18.280+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:18.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:18.280+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:18.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:18.281+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:18.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:18.570+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:18.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:18.570+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:18.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:18.570+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:18.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:18 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:18.571+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:19.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:19.295+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:19.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:19.295+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:19.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:19.295+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:19.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:19.295+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:19.592 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:19.359+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:19.592 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:19.359+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:19.592 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:19.359+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:19.592 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:19.359+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:19.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:19.590+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:19.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:19.590+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:19.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:19.590+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:19.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:19 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:19.591+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:20.547 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:20.394+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:20.548 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:20.394+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:20.548 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:20.394+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:20.548 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:20.395+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:20.548 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:20.277+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:20.548 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:20.277+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:20.548 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:20.277+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:20.549 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:20.278+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:20.549 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:20 smithi059 systemd[1]: Stopping Ceph osd.0 for 3fc07a38-b9cb-11ee-95b1-87774f69a715... 2024-01-23T09:29:20.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:20.713+0000 7f707dd37700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-23T09:29:20.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:20.713+0000 7f707dd37700 -1 osd.0 47 *** Got signal Terminated *** 2024-01-23T09:29:20.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:20.713+0000 7f707dd37700 -1 osd.0 47 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-23T09:29:20.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:20.619+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:20.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:20.619+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:20.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:20.619+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:20.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:20 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:20.619+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:21.576 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:21.384+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:21.576 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:21.384+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:21.576 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:21.384+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:21.577 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:21.385+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:21.577 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:21.261+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:21.577 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:21.261+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:21.577 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:21.261+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:21.577 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:21.262+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:21.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:21.576+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:21.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:21.576+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:21.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:21.576+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:21.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:21 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:21.577+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:22.579 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:22.355+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:22.580 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:22.355+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:22.580 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:22.355+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:22.580 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:22.356+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:22.580 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:22.257+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:22.580 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:22.257+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:22.581 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:22.257+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:22.581 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:22.258+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:22.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:22.579+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:22.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:22.579+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:22.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:22.579+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:22.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:22 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:22.580+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:23.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:23.219+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:23.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:23.219+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:23.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:23.219+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:23.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:23.220+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:23.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:23.383+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:23.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:23.383+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:23.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:23.383+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:23.477 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:23.384+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:23.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:23.612+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:23.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:23.612+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:23.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:23.612+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:23.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:23 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:23.612+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:24.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:24.183+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:24.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:24.183+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:24.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:24.183+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:24.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:24.184+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:24.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:24.342+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:24.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:24.342+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:24.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:24.342+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:24.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:24.343+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:24.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:24.656+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:24.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:24.656+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:24.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:24.656+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:24.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:24 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:24.657+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:25.474 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:25.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:25.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:25.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:25.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:25.197+0000 7f7079333700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:37.569808+0000 (oldest deadline 2024-01-23T09:23:57.569808+0000) 2024-01-23T09:29:25.475 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0[46708]: 2024-01-23T09:29:25.198+0000 7f7079333700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24317.0:4878 3.12 3:4dce097f:::1000000430d.00000000:head [write 0~1415 in=1415b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:25.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:25.363+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:25.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:25.363+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:25.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:25.363+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:25.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:25.364+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:25.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:25.623+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:25.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:25.623+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:25.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:25.623+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:25.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:25 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:25.623+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:26.626 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:26 smithi059 podman[85350]: 2024-01-23 09:29:26.307326487 +0000 UTC m=+5.722674784 container died 11074a2f09abfd7e3d40cf6f89e52c7487be45f457148d11b3a9f551de21f248 (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=quincy-b824e5b, io.buildah.version=1.31.3, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240118, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e) 2024-01-23T09:29:26.626 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:26.334+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:26.626 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:26.334+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:26.627 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:26.334+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:26.627 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:26.335+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:26.965 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:26 smithi059 podman[85350]: 2024-01-23 09:29:26.738982365 +0000 UTC m=+6.154330659 container cleanup 11074a2f09abfd7e3d40cf6f89e52c7487be45f457148d11b3a9f551de21f248 (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-b824e5b, io.buildah.version=1.31.3, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240118, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_BRANCH=HEAD) 2024-01-23T09:29:26.966 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:26 smithi059 bash[85350]: ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0 2024-01-23T09:29:26.966 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:26.662+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:26.966 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:26.662+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:26.967 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:26.662+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:26.967 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:26 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:26.663+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:27.224 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:26 smithi059 podman[85382]: 2024-01-23 09:29:26.965450509 +0000 UTC m=+0.662126648 container remove 11074a2f09abfd7e3d40cf6f89e52c7487be45f457148d11b3a9f551de21f248 (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240118, io.buildah.version=1.31.3, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=quincy-b824e5b, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS) 2024-01-23T09:29:27.595 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:27 smithi059 podman[85475]: 2024-01-23 09:29:27.316322343 +0000 UTC m=+0.017518925 image pull 61b360e6ff650187c031550cf8bb1c0d3e28fc466cffb29cb06fa0da2544d680 quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001 2024-01-23T09:29:27.595 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:27 smithi059 podman[85475]: 2024-01-23 09:29:27.449770707 +0000 UTC m=+0.150967285 container create 94cccb382dd3c80c5682bb3808e4266d8ab96ad28c8fd1ed24446a12f0e6d60d (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=quincy-b824e5b, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, io.buildah.version=1.31.3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240118, GIT_CLEAN=True) 2024-01-23T09:29:27.596 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:27.330+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:27.596 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:27.330+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:27.596 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:27.330+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:27.596 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:27.331+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:27.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:27.687+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:27.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:27.687+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:27.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:27.687+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:27.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:27 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:27.688+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:27.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:27 smithi059 podman[85475]: 2024-01-23 09:29:27.694108492 +0000 UTC m=+0.395305067 container init 94cccb382dd3c80c5682bb3808e4266d8ab96ad28c8fd1ed24446a12f0e6d60d (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.build-date=20240118, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=quincy-b824e5b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.31.3, org.label-schema.name=CentOS Stream 8 Base Image) 2024-01-23T09:29:27.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:27 smithi059 podman[85475]: 2024-01-23 09:29:27.697283787 +0000 UTC m=+0.398480370 container start 94cccb382dd3c80c5682bb3808e4266d8ab96ad28c8fd1ed24446a12f0e6d60d (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.build-date=20240118, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=quincy-b824e5b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.31.3, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-01-23T09:29:27.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:27 smithi059 podman[85475]: 2024-01-23 09:29:27.785811745 +0000 UTC m=+0.487008329 container attach 94cccb382dd3c80c5682bb3808e4266d8ab96ad28c8fd1ed24446a12f0e6d60d (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0-deactivate, RELEASE=quincy-b824e5b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.31.3, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240118, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True) 2024-01-23T09:29:28.346 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:27 smithi059 podman[85475]: 2024-01-23 09:29:27.982678422 +0000 UTC m=+0.683874998 container died 94cccb382dd3c80c5682bb3808e4266d8ab96ad28c8fd1ed24446a12f0e6d60d (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.31.3, org.label-schema.build-date=20240118, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=quincy-b824e5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-01-23T09:29:28.707 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:28.351+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:28.707 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:28.351+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:28.707 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:28.351+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:28.707 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:28.351+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:28.974 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:28 smithi059 systemd[1]: Stopping Ceph osd.1 for 3fc07a38-b9cb-11ee-95b1-87774f69a715... 2024-01-23T09:29:28.974 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:28 smithi059 podman[85501]: 2024-01-23 09:29:28.710017318 +0000 UTC m=+0.721445066 container remove 94cccb382dd3c80c5682bb3808e4266d8ab96ad28c8fd1ed24446a12f0e6d60d (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=quincy-b824e5b, ceph=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240118, org.label-schema.name=CentOS Stream 8 Base Image) 2024-01-23T09:29:28.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:28 smithi059 systemd[1]: ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@osd.0.service: Deactivated successfully. 2024-01-23T09:29:28.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:28 smithi059 systemd[1]: Stopped Ceph osd.0 for 3fc07a38-b9cb-11ee-95b1-87774f69a715. 2024-01-23T09:29:28.975 INFO:journalctl@ceph.osd.0.smithi059.stdout:Jan 23 09:29:28 smithi059 systemd[1]: ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@osd.0.service: Consumed 18min 20.727s CPU time. 2024-01-23T09:29:28.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:28.705+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:28.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:28.705+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:28.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:28.705+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:28.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:28 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:28.706+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:29.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:29.175+0000 7f27e1bbd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-23T09:29:29.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:29.175+0000 7f27e1bbd700 -1 osd.1 47 *** Got signal Terminated *** 2024-01-23T09:29:29.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:29.175+0000 7f27e1bbd700 -1 osd.1 47 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-23T09:29:29.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:29.397+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:29.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:29.397+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:29.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:29.397+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:29.475 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:29.398+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:30.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:29.736+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:30.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:29.736+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:30.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:29.736+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:30.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:29 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:29.737+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:30.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:30.359+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:30.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:30.359+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:30.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:30.359+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:30.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:30.360+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:31.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:30.785+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:31.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:30.785+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:31.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:30.785+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:31.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:30 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:30.786+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:31.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:31.331+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:31.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:31.331+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:31.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:31.331+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:31.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:31.331+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:32.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:31.830+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:32.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:31.830+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:32.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:31.830+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:32.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:31 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:31.831+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:32.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:32.369+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:32.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:32.369+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:32.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:32.369+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:32.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:32.370+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:33.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:32.788+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:33.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:32.788+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:33.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:32.788+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:33.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:32 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:32.788+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:33.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:33.353+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:33.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:33.353+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:33.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:33.353+0000 7f27df9be700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:47.939602+0000 (oldest deadline 2024-01-23T09:24:07.939602+0000) 2024-01-23T09:29:33.725 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1[53913]: 2024-01-23T09:29:33.353+0000 7f27df9be700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.14480.14:463 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=189917b,omap-rm-keys in=10519b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2024-01-23T09:29:34.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:33.835+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:34.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:33.835+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:34.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:33.835+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:34.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:33 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:33.836+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:35.086 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:34 smithi059 podman[85578]: 2024-01-23 09:29:34.818784458 +0000 UTC m=+5.767230955 container died efe49a487df246c18c2f7aa04dd4d8726fa20e1aea2dc898cc899fb5314dc05c (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.31.3, org.label-schema.build-date=20240118, RELEASE=quincy-b824e5b, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-01-23T09:29:35.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:34.847+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:35.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:34.847+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:35.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:34.847+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:35.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:34 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:34.847+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:35.469 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:35 smithi059 podman[85578]: 2024-01-23 09:29:35.168454074 +0000 UTC m=+6.116900573 container cleanup efe49a487df246c18c2f7aa04dd4d8726fa20e1aea2dc898cc899fb5314dc05c (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=quincy-b824e5b, GIT_BRANCH=HEAD, io.buildah.version=1.31.3, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_CLEAN=True, org.label-schema.build-date=20240118) 2024-01-23T09:29:35.469 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:35 smithi059 bash[85578]: ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1 2024-01-23T09:29:35.724 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:35 smithi059 podman[85597]: 2024-01-23 09:29:35.469568819 +0000 UTC m=+0.649475804 container remove efe49a487df246c18c2f7aa04dd4d8726fa20e1aea2dc898cc899fb5314dc05c (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240118, RELEASE=quincy-b824e5b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-01-23T09:29:36.097 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:35.825+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:36.097 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:35.825+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:36.098 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:35.825+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:36.098 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:35 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:35.826+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:36.098 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:35 smithi059 podman[85690]: 2024-01-23 09:29:35.685050673 +0000 UTC m=+0.018041874 image pull 61b360e6ff650187c031550cf8bb1c0d3e28fc466cffb29cb06fa0da2544d680 quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001 2024-01-23T09:29:36.098 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:35 smithi059 podman[85690]: 2024-01-23 09:29:35.795521387 +0000 UTC m=+0.128512625 container create eae47b2eace6160c9987a8a0d6c8a74071eca12059ec033abe93daed85b1df20 (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=quincy-b824e5b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240118, GIT_CLEAN=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, io.buildah.version=1.31.3) 2024-01-23T09:29:36.098 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:36 smithi059 podman[85690]: 2024-01-23 09:29:36.014748564 +0000 UTC m=+0.347739763 container init eae47b2eace6160c9987a8a0d6c8a74071eca12059ec033abe93daed85b1df20 (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240118, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=quincy-b824e5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, io.buildah.version=1.31.3, CEPH_POINT_RELEASE=) 2024-01-23T09:29:36.098 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:36 smithi059 podman[85690]: 2024-01-23 09:29:36.017869583 +0000 UTC m=+0.350860788 container start eae47b2eace6160c9987a8a0d6c8a74071eca12059ec033abe93daed85b1df20 (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, ceph=True, io.buildah.version=1.31.3, org.label-schema.build-date=20240118, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=quincy-b824e5b, GIT_BRANCH=HEAD) 2024-01-23T09:29:36.098 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:36 smithi059 podman[85690]: 2024-01-23 09:29:36.097861504 +0000 UTC m=+0.430852700 container attach eae47b2eace6160c9987a8a0d6c8a74071eca12059ec033abe93daed85b1df20 (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, io.buildah.version=1.31.3, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-b824e5b, org.label-schema.build-date=20240118, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e) 2024-01-23T09:29:36.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:36 smithi059 podman[85690]: 2024-01-23 09:29:36.306863076 +0000 UTC m=+0.639854275 container died eae47b2eace6160c9987a8a0d6c8a74071eca12059ec033abe93daed85b1df20 (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1-deactivate, io.buildah.version=1.31.3, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, RELEASE=quincy-b824e5b, org.label-schema.build-date=20240118, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e) 2024-01-23T09:29:36.892 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:36.788+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:36.892 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:36.788+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:36.892 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:36.788+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:36.893 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:36 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:36.788+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:37.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:37 smithi059 systemd[1]: Stopping Ceph osd.2 for 3fc07a38-b9cb-11ee-95b1-87774f69a715... 2024-01-23T09:29:37.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:36 smithi059 podman[85716]: 2024-01-23 09:29:36.892850741 +0000 UTC m=+0.580488518 container remove eae47b2eace6160c9987a8a0d6c8a74071eca12059ec033abe93daed85b1df20 (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=quincy-b824e5b, CEPH_POINT_RELEASE=, io.buildah.version=1.31.3, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240118, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e) 2024-01-23T09:29:37.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:36 smithi059 systemd[1]: ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@osd.1.service: Deactivated successfully. 2024-01-23T09:29:37.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:36 smithi059 systemd[1]: Stopped Ceph osd.1 for 3fc07a38-b9cb-11ee-95b1-87774f69a715. 2024-01-23T09:29:37.225 INFO:journalctl@ceph.osd.1.smithi059.stdout:Jan 23 09:29:36 smithi059 systemd[1]: ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@osd.1.service: Consumed 19min 50.556s CPU time. 2024-01-23T09:29:37.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:37.344+0000 7fc62451e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-23T09:29:37.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:37.344+0000 7fc62451e700 -1 osd.2 47 *** Got signal Terminated *** 2024-01-23T09:29:37.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:37.344+0000 7fc62451e700 -1 osd.2 47 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-23T09:29:38.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:37.762+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:38.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:37.763+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:38.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:37.763+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:38.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:37 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:37.763+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:39.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:38.758+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:39.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:38.758+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:39.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:38.758+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:39.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:38 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:38.759+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:40.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:39.776+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:40.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:39.776+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:40.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:39.776+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:40.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:39 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:39.776+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:41.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:40.768+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:41.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:40.768+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:41.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:40.768+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:41.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:40 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:40.768+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:42.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:41.746+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6804 osd.3 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:42.224 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:41.746+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6812 osd.4 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:42.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:41.746+0000 7fc61db16700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.137:6820 osd.5 ever on either front or back, first ping sent 2024-01-23T09:23:33.581369+0000 (oldest deadline 2024-01-23T09:23:53.581369+0000) 2024-01-23T09:29:42.225 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:41 smithi059 ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2[61659]: 2024-01-23T09:29:41.746+0000 7fc61db16700 -1 osd.2 47 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.24317.0:4875 3.8 3:11b08ac7:::10000003e8d.00000000:head [write 0~5016 in=5016b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-01-23T09:29:43.037 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:42 smithi059 podman[85795]: 2024-01-23 09:29:42.759981722 +0000 UTC m=+5.507585394 container died 0e2f974c19f06db0efe38abfef2075310c8a1e9078f274dc06b8102de5d62afe (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=quincy-b824e5b, org.label-schema.build-date=20240118, ceph=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.vendor=CentOS, io.buildah.version=1.31.3) 2024-01-23T09:29:43.396 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:43 smithi059 podman[85795]: 2024-01-23 09:29:43.136978603 +0000 UTC m=+5.884582265 container cleanup 0e2f974c19f06db0efe38abfef2075310c8a1e9078f274dc06b8102de5d62afe (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-b824e5b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240118, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.31.3, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-01-23T09:29:43.396 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:43 smithi059 bash[85795]: ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2 2024-01-23T09:29:43.690 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:43 smithi059 podman[85812]: 2024-01-23 09:29:43.396749261 +0000 UTC m=+0.633093325 container remove 0e2f974c19f06db0efe38abfef2075310c8a1e9078f274dc06b8102de5d62afe (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=quincy-b824e5b, org.label-schema.vendor=CentOS, io.buildah.version=1.31.3, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240118, org.label-schema.license=GPLv2) 2024-01-23T09:29:43.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:43 smithi059 podman[85904]: 2024-01-23 09:29:43.593014673 +0000 UTC m=+0.017491224 image pull 61b360e6ff650187c031550cf8bb1c0d3e28fc466cffb29cb06fa0da2544d680 quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001 2024-01-23T09:29:43.974 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:43 smithi059 podman[85904]: 2024-01-23 09:29:43.705821239 +0000 UTC m=+0.130297831 container create 6c66fc4b62d3742df78ac8736b0a77c7496d53647653fc24ba3dcbe9f8b63ab3 (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2-deactivate, RELEASE=quincy-b824e5b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.31.3, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240118, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-01-23T09:29:43.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:43 smithi059 podman[85904]: 2024-01-23 09:29:43.916354626 +0000 UTC m=+0.340831178 container init 6c66fc4b62d3742df78ac8736b0a77c7496d53647653fc24ba3dcbe9f8b63ab3 (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.31.3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.build-date=20240118, RELEASE=quincy-b824e5b) 2024-01-23T09:29:43.975 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:43 smithi059 podman[85904]: 2024-01-23 09:29:43.919437349 +0000 UTC m=+0.343913898 container start 6c66fc4b62d3742df78ac8736b0a77c7496d53647653fc24ba3dcbe9f8b63ab3 (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240118, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.schema-version=1.0, io.buildah.version=1.31.3, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=quincy-b824e5b, org.label-schema.vendor=CentOS) 2024-01-23T09:29:44.474 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:43 smithi059 podman[85904]: 2024-01-23 09:29:43.999653012 +0000 UTC m=+0.424129567 container attach 6c66fc4b62d3742df78ac8736b0a77c7496d53647653fc24ba3dcbe9f8b63ab3 (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240118, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=quincy-b824e5b, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.31.3, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.vendor=CentOS) 2024-01-23T09:29:44.475 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:44 smithi059 podman[85904]: 2024-01-23 09:29:44.202323468 +0000 UTC m=+0.626800027 container died 6c66fc4b62d3742df78ac8736b0a77c7496d53647653fc24ba3dcbe9f8b63ab3 (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.31.3, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=quincy-b824e5b, org.label-schema.build-date=20240118, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-01-23T09:29:45.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:44 smithi059 podman[85930]: 2024-01-23 09:29:44.744386716 +0000 UTC m=+0.536498201 container remove 6c66fc4b62d3742df78ac8736b0a77c7496d53647653fc24ba3dcbe9f8b63ab3 (image=quay.ceph.io/ceph-ci/ceph@sha256:1ea0178abdd78fd629cf2de4f742b987c9e42ba8d2675a2894cf54b5ea1e7001, name=ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.31.3, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240118, GIT_BRANCH=HEAD, RELEASE=quincy-b824e5b, GIT_CLEAN=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, maintainer=Guillaume Abrioux ) 2024-01-23T09:29:45.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:44 smithi059 systemd[1]: ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@osd.2.service: Deactivated successfully. 2024-01-23T09:29:45.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:44 smithi059 systemd[1]: Stopped Ceph osd.2 for 3fc07a38-b9cb-11ee-95b1-87774f69a715. 2024-01-23T09:29:45.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Jan 23 09:29:44 smithi059 systemd[1]: ceph-3fc07a38-b9cb-11ee-95b1-87774f69a715@osd.2.service: Consumed 20.978s CPU time. 2024-01-23T09:29:47.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:29:47.840 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:29:50.908 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:29:59.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:29:59.910 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:30:02.972 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:30:14.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:30:14.975 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:30:18.044 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:30:21.045 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:30:21.046 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:30:21.116 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:30:21.116 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-23T09:30:21.117 INFO:tasks.cephadm:Teardown complete 2024-01-23T09:30:21.117 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/cephadm.py", line 1591, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/cephadm.py", line 1845, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/cephadm.py", line 1848, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/cephadm.py", line 1724, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/cephadm.py", line 180, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/cephadm.py", line 348, in _rm_cluster ctx.cluster.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi137 2024-01-23T09:30:21.141 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-23T09:30:21.154 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-01-23T09:30:21.171 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-23T09:30:21.172 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-23T09:30:21.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:30:21.185 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:30:24.252 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:30:33.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:30:33.254 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:30:36.316 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:30:48.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:30:48.318 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:30:51.388 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:30:54.389 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:30:54.390 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:30:54.460 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:30:54.460 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-23T09:30:54.513 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi137 2024-01-23T09:30:54.535 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-23T09:30:54.546 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-23T09:30:54.547 DEBUG:teuthology.orchestra.run.smithi059:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-23T09:30:54.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:30:54.556 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:30:54.570 INFO:teuthology.orchestra.run.smithi059.stderr:bash: line 1: ntpq: command not found 2024-01-23T09:30:54.574 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-23T09:30:54.574 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2024-01-23T09:30:54.574 INFO:teuthology.orchestra.run.smithi059.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 57 +839us[ +839us] +/- 72ms 2024-01-23T09:30:54.575 INFO:teuthology.orchestra.run.smithi059.stdout:^* hv02.front.sepia.ceph.com 3 8 377 186 -758us[ -734us] +/- 79ms 2024-01-23T09:30:54.575 INFO:teuthology.orchestra.run.smithi059.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 59 +189us[ +189us] +/- 87ms 2024-01-23T09:30:54.575 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-01-23T09:30:57.632 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:31:06.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:31:06.634 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:31:09.692 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:31:21.693 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:31:21.694 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:31:24.764 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:31:27.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:31:27.766 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:31:27.836 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:31:27.836 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-23T09:31:27.837 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi137 2024-01-23T09:31:27.852 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-23T09:31:27.863 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-23T09:31:27.864 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-23T09:31:27.950 DEBUG:teuthology.orchestra.run.smithi059:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-23T09:31:28.037 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:31:28.037 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:31:31.100 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:31:40.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:31:40.104 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:31:43.164 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:31:55.165 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:31:55.166 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:31:58.240 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:32:01.241 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:32:01.242 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:32:01.312 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:32:01.312 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-23T09:32:01.313 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi137 2024-01-23T09:32:01.321 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-23T09:32:01.333 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-23T09:32:01.356 INFO:teuthology.task.internal:Duration was 3812.273714 seconds 2024-01-23T09:32:01.356 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-23T09:32:01.395 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-23T09:32:01.396 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-23T09:32:01.402 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:32:01.403 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:32:01.450 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-23T09:32:04.476 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:32:13.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:32:13.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:32:16.544 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:32:28.545 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:32:28.546 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:32:31.616 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:32:34.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:32:34.618 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:32:34.684 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:32:34.684 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-23T09:32:34.685 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi137 2024-01-23T09:32:34.701 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-23T09:32:34.719 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-23T09:32:34.719 DEBUG:teuthology.orchestra.run.smithi059:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-23T09:32:34.757 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:32:34.757 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:32:37.820 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:32:46.821 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:32:46.822 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:32:49.888 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:33:01.889 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:33:01.890 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:33:04.960 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:33:07.961 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:33:07.962 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:33:08.028 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:33:08.028 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-23T09:33:08.029 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi137 2024-01-23T09:33:08.045 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-23T09:33:08.058 DEBUG:teuthology.orchestra.run.smithi059:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-23T09:33:08.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:33:08.067 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:33:08.093 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = core 2024-01-23T09:33:11.132 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:33:20.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:33:20.134 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:33:23.196 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:33:35.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:33:35.199 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:33:38.268 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:33:41.269 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:33:41.270 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:33:41.340 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:33:41.340 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-23T09:33:41.341 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi137 2024-01-23T09:33:41.358 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-23T09:33:41.371 INFO:teuthology.task.internal:Transferring archived files... 2024-01-23T09:33:41.372 DEBUG:teuthology.misc:Transferring archived files from smithi059:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/cbodley-2024-01-22_19:46:51-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7527396/remote/smithi059 2024-01-23T09:33:41.372 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-23T09:33:41.414 DEBUG:teuthology.misc:Transferring archived files from smithi137:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/cbodley-2024-01-22_19:46:51-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7527396/remote/smithi137 2024-01-23T09:33:41.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:33:41.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:33:44.476 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:33:53.477 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:33:53.478 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:33:56.544 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:34:08.545 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:34:08.546 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:34:11.612 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:34:14.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:34:14.614 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:34:14.688 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:34:14.688 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-23T09:34:14.689 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi137 2024-01-23T09:34:14.705 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-23T09:34:14.718 INFO:teuthology.task.internal:Not uploading archives. 2024-01-23T09:34:14.718 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-23T09:34:14.750 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-23T09:34:14.751 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-23T09:34:14.758 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:34:14.759 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:34:14.774 INFO:teuthology.orchestra.run.smithi059.stdout: 265304 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 23 09:30 /home/ubuntu/cephtest 2024-01-23T09:34:14.774 INFO:teuthology.orchestra.run.smithi059.stdout: 265305 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Jan 23 09:33 /home/ubuntu/cephtest/archive 2024-01-23T09:34:14.775 INFO:teuthology.orchestra.run.smithi059.stdout: 265465 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 23 08:28 /home/ubuntu/cephtest/archive/syslog 2024-01-23T09:34:14.775 INFO:teuthology.orchestra.run.smithi059.stdout: 265536 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Jan 23 08:28 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-23T09:34:14.775 INFO:teuthology.orchestra.run.smithi059.stdout: 265496 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Jan 23 08:28 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-23T09:34:14.775 INFO:teuthology.orchestra.run.smithi059.stdout: 530355 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 23 09:31 /home/ubuntu/cephtest/archive/audit 2024-01-23T09:34:14.775 INFO:teuthology.orchestra.run.smithi059.stdout: 530357 176 -rw------- 1 ubuntu ubuntu 177679 Jan 23 09:31 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-01-23T09:34:14.775 INFO:teuthology.orchestra.run.smithi059.stdout: 265333 360 -rwxr-xr-x 1 ubuntu ubuntu 367151 Jan 23 08:41 /home/ubuntu/cephtest/cephadm 2024-01-23T09:34:14.776 INFO:teuthology.orchestra.run.smithi059.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-01-23T09:34:17.820 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:34:26.821 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:34:26.822 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:34:29.884 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:34:41.886 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:34:41.886 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:34:44.956 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:34:47.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-23T09:34:47.958 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-23T09:34:48.028 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-23T09:34:48.028 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-23T09:34:48.029 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_cbodley_ceph_9db6bac5568c8be6cfd98da20bd2b62582f0776f/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi137 2024-01-23T09:34:48.045 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-23T09:34:48.058 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-23T09:34:48.088 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2024-01-23T09:34:48.089 INFO:teuthology.nuke:Checking targets against current locks 2024-01-23T09:34:48.122 DEBUG:teuthology.nuke:shortname: smithi059 2024-01-23T09:34:48.122 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-23T09:34:48.147 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/cbodley-2024-01-22_19:46:51-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7527396', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-23 08:20:06.787289', 'locked_by': 'scheduled_cbodley@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFnPecS4qMynd+IIA/wWoEbFcLUsPHz/OCadDhe+id0B+IvoB2IfSE2xoqvZNe3ZL856Tnc42YnsRd/eOSxzGIc='} 2024-01-23T09:34:48.151 DEBUG:teuthology.nuke:shortname: smithi137 2024-01-23T09:34:48.151 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-23T09:34:48.174 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/cbodley-2024-01-22_19:46:51-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7527396', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-23 08:20:06.785801', 'locked_by': 'scheduled_cbodley@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMzyUqHuzAHFsSqL7zeDd6gRwxbOLdPxiEfr+GvgX1i0VlB597qQ/Sg8Xn1J9zN3+m7vLflnL1OsnpeMFpcMQ0I='} 2024-01-23T09:34:48.209 INFO:teuthology.orchestra.console.smithi059:Power off 2024-01-23T09:34:48.209 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-23T09:34:48.236 DEBUG:teuthology.orchestra.console.smithi059:power off output: Chassis Power Control: Down/Off 2024-01-23T09:34:48.236 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-23T09:34:48.254 INFO:teuthology.orchestra.console.smithi137:Power off 2024-01-23T09:34:48.254 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-23T09:34:48.264 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is on 2024-01-23T09:34:48.277 DEBUG:teuthology.orchestra.console.smithi137:power off output: Chassis Power Control: Down/Off 2024-01-23T09:34:48.277 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-23T09:34:48.300 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2024-01-23T09:34:52.265 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-23T09:34:52.301 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-23T09:34:52.375 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is on 2024-01-23T09:34:52.412 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2024-01-23T09:34:56.380 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-23T09:34:56.413 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-23T09:34:56.491 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2024-01-23T09:34:56.524 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is off 2024-01-23T09:34:56.592 INFO:teuthology.orchestra.console.smithi059:Power off completed 2024-01-23T09:34:56.625 INFO:teuthology.orchestra.console.smithi137:Power off completed 2024-01-23T09:34:56.731 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 3812.2737143039703 failure_reason: 'Command failed on smithi059 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fc07a38-b9cb-11ee-95b1-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c \''ceph versions | jq -e \''"\''"\''.mgr | keys\''"\''"\'' | grep $sha1\''''' flavor: default owner: scheduled_cbodley@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=cab3e883645d4de3bc39e5fd1350d988 status: fail success: false 2024-01-23T09:34:56.731 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-23T09:34:56.816 INFO:teuthology.run:FAIL